Message Logger: removed forums load, stay smooth and cached

- Topics load fully again: the bottom-bar cleanup no longer hides the
  message list; each page fetches a full window from the server and
  answers only what the app asked for.
- Reopened chats and topics come from memory (pages, logged copies,
  downloaded media); pages are also kept on the device for restarts and
  server timeouts, with one retry before falling back.
- Media downloads run in one background queue per chat instead of
  blocking page answers or spawning a thread per page.
- The deleted badge is drawn in DialogCell.drawAvatarOverlays, under the
  avatar's own transform, so it follows the avatar through transitions;
  its paints are built once and the hot hooks skip reflection.
- Senders keep their last known profile (name, username, photo
  reference) when the app no longer has them (people.py).
- Removed chats are searchable, show "deleted chat" only in place of the
  loading subtitle, hide the join bar, order topics by activity, and
  have no unread counters.
- Uninstall takes removed chats out on the UI thread.

README.md

@@ -22,9 +22,19 @@ the device, so a chat opened again, even offline, shows them at once.
   was closed are confirmed with Telegram and restored from the logger copy.
 - **Keep your own deletions.** The delete dialog gets a *Keep message for myself* checkbox: ticked,
   the deletion goes to Telegram but the message stays on this device, marked deleted.
-  Unticked (the default), it is deleted as usual and never restored from the logger here.
+  Unticked (the default), it is deleted as usual and hidden, so it isn't restored from the
-  Deleting a message that is already deleted shows a plain dialog that removes it from view
+  logger. Deleting a message that is already deleted shows a plain dialog that hides it.
-  on this device only.
+- **Hidden messages.** Hidden deleted messages are marked on the server, so every client
+  skips them. *Reveal hidden messages* in the chat's menu (Plugins) brings them back until
+  tapped again; a revealed one has *Unhide* in its menu, and selecting several shows an
+  Unhide button in the selection bar.
+- **Removed chats.** Chats an account lost (left, kicked, deleted by you or the other side)
+  stay in the chat list with their logged messages, read-only and all marked deleted, until
+  the chat is back. Chats you delete yourself aren't listed; deleting a listed one hides it.
+  *Show removed chats* in the settings turns this off.
+- **Old chat photos.** Chat photo changes are posted into the chat's log topic. A removed chat
+  shows its latest logged photo, and a chat's photos also list the logged ones it no longer
+  has: translucent, with a trash icon, in the photo viewer's previews.
 - **Edit history.** *Edit history* in the message menu (right above Forward, for edited
   messages) opens every version as a mock chat, also for the logger groups' own posts, with
   an *Open topic* button for the history topic in the activity group.
@@ -36,7 +46,7 @@ the device, so a chat opened again, even offline, shows them at once.
   going back returns to your account.
 - **Reset marks.** *Reset deleted/edited marks* in the chat's menu (Plugins) forgets that
   chat's cached marks and checks it with the server again; the same row in the plugin
-  settings does it for every chat. Messages you removed from view stay hidden.
+  settings does it for every chat. Hidden messages stay hidden.
 - **Cache lifetime.** Cached answers are kept for 1, 7, 30, 90 or 365 days, or forever
   (30 by default).
 

plugin.json

@@ -3,7 +3,7 @@
   "name": "Message Logger",
   "description": "Shows deleted and edited messages and their edit history from your telegram-panel message logger.",
   "author": "@lyuksovannyy",
-  "version": "0.1.2",
+  "version": "0.1.7",
   "icon": "exteraGramPluginIcons/4",
   "app_version": ">=12.5.1",
   "sdk_version": ">=1.4.4.3",

src/api.py

@@ -99,6 +99,18 @@ class Api:
                 out.append((token[:10], exc))
         return out
 
+    def masters(self):
+        """Telegram ids of the tokens' masters (from /me), cached 10 minutes."""
+        cached = getattr(self, "_masters", None)
+        if cached is not None and time.monotonic() - cached[0] < 600:
+            return cached[1]
+        found = []
+        for _prefix, me in self.me():
+            if isinstance(me, dict) and me.get("master_tg_id"):
+                found.append(int(me["master_tg_id"]))
+        self._masters = (time.monotonic(), found)
+        return found
+
     def sync(self, account_id, chat_id, min_id, max_id, present_ids):
         body = {
             "account_id": account_id,
@@ -152,3 +164,72 @@ class Api:
             for _token, result in self._each("POST", "/deletions", json=body):
                 marked += int((result or {}).get("marked", 0))
         return marked
+
+    def hidden(self, account_id, chat_id, message_ids, hidden):
+        """Hide messages from the chat for every client (or show them again);
+        True once every master took it."""
+        ids = sorted(set(int(i) for i in message_ids))
+        for start in range(0, len(ids), 500):
+            body = {
+                "account_id": account_id,
+                "chat_id": chat_id,
+                "message_ids": ids[start : start + 500],
+                "hidden": bool(hidden),
+            }
+            for _token, _result in self._each("POST", "/hidden", json=body):
+                pass
+        return True
+
+    # removed chats
+
+    def chat_removed(self, account_id, chat_id, own):
+        body = {"account_id": account_id, "chat_id": chat_id, "own": bool(own)}
+        known = False
+        for _token, result in self._each("POST", "/chats/removed", json=body):
+            known = known or bool((result or {}).get("known"))
+        return known
+
+    def chat_hidden(self, account_id, chat_id, hidden):
+        body = {"account_id": account_id, "chat_id": chat_id, "hidden": bool(hidden)}
+        for _token, _result in self._each("POST", "/chats/hidden", json=body):
+            pass
+
+    def removed_chats(self, account_id):
+        """The account's removed chats, each master's answer merged (the
+        first one listing a chat wins), or None when no master answered."""
+        chats, any_result = {}, False
+        for _token, result in self._each(
+            "GET", "/chats/removed", params={"account_id": account_id}
+        ):
+            if result is None:
+                continue
+            any_result = True
+            for chat in result.get("chats") or []:
+                chat["master_tg_id"] = result.get("master_tg_id")
+                chats.setdefault(int(chat["chat_id"]), chat)
+        return list(chats.values()) if any_result else None
+
+    def history(self, account_id, chat_id, before=0, limit=100, topic_id=None):
+        """Logged messages older than `before`, newest first. For a forum
+        topic, pass its source topic id; 0 selects General."""
+        params = {"account_id": account_id, "before": int(before), "limit": int(limit)}
+        if topic_id is not None:
+            params["topic_id"] = int(topic_id)
+        for _token, result in self._each(
+            "GET", f"/chats/{chat_id}/history", params=params
+        ):
+            if result and result.get("messages"):
+                return result
+        return None
+
+    def avatars(self, account_id, chat_id):
+        """Chat photos logged for a chat, newest first, every master's."""
+        out = []
+        for _token, result in self._each(
+            "GET", f"/chats/{chat_id}/avatars", params={"account_id": account_id}
+        ):
+            for avatar in (result or {}).get("avatars") or []:
+                avatar["master_tg_id"] = (result or {}).get("master_tg_id")
+                out.append(avatar)
+        out.sort(key=lambda a: a.get("date") or "", reverse=True)
+        return out

src/config.py

@@ -7,6 +7,7 @@ RESTORE_MISSING = "restore_missing"
 REPORT_DELETIONS = "report_deletions"
 DELETED_ALPHA = "deleted_alpha"
 CACHE_DAYS = "cache_days"
+SHOW_REMOVED_CHATS = "show_removed_chats"
 
 DEFAULTS = {
     SERVER_URL: "",
@@ -16,6 +17,7 @@ DEFAULTS = {
     REPORT_DELETIONS: True,
     DELETED_ALPHA: 0,  # index into ALPHA_CHOICES
     CACHE_DAYS: 2,  # index into CACHE_DAYS_CHOICES
+    SHOW_REMOVED_CHATS: True,
 }
 
 # Opacity of deleted messages: well visible, marked by the trash icon.

src/hooks/avatars.py

@@ -0,0 +1,574 @@
+"""Chat photos the logger posted (service posts in the chat's log topic, see
+telegram-panel `avatars`), read from those posts.
+
+- A removed chat shows its latest logged photo (hooks/removed_chats.py).
+- A chat's photos (MessagesController.getDialogPhotos: DialogPhotos `photos`,
+  shown by the profile's gallery and the photo viewer, both redrawn on
+  NotificationCenter.dialogPhotosUpdate) get the logged photos the chat no
+  longer has, after its own. The photo viewer's thumbnails of those
+  (GroupedPhotosListView.onDraw) are translucent like deleted messages, with
+  a red trash icon.
+
+A post is read and its photo downloaded through an account that can read
+the log group; the app's file cache is shared by its accounts, so the photo
+then shows for any of them.
+"""
+
+import threading
+import time
+
+from base_plugin import MethodHook
+
+import diag
+import media_share
+import peers
+from api import ApiError
+from jutil import field, hook_methods, in_background, java_class, java_list, on_ui
+from views.history import _epoch
+
+CONTROLLER = "org.telegram.messenger.MessagesController"
+THUMBS = "org.telegram.ui.Components.GroupedPhotosListView"
+VIEWER = "org.telegram.ui.PhotoViewer"
+GALLERY = "org.telegram.ui.Components.ProfileGalleryView"
+OVERLAYS = "org.telegram.ui.ProfileActivity$OverlaysView"
+_MARK_TAG = "message_logger_trash"
+NOTIFICATIONS = "org.telegram.messenger.NotificationCenter"
+# A logged photo whose id isn't Telegram's (logged by a bot) is the chat's
+# own when one of the chat's photos was set this close to its post.
+_SAME_PHOTO_SECONDS = 10
+# A chat's logged photos are asked for again after this long.
+_FRESH_SECONDS = 600
+_TELL_SECONDS = 3
+
+
+def _photo_of(copy):
+    """The photo of a logger post (a photo message, or the first photo of a
+    rich message)."""
+    if copy is None:
+        return None
+    photo = getattr(getattr(copy, "media", None), "photo", None)
+    if photo is not None and getattr(photo, "sizes", None) is not None:
+        return photo
+    rich = getattr(copy, "rich_message", None)
+    photos = getattr(rich, "photos", None) if rich is not None else None
+    if photos is not None and photos.size():
+        return photos.get(0)
+    return None
+
+
+def logged_photo(account, avatar, masters=()):
+    """(photo, reader slot, post) of a logged chat photo, read through a
+    member of its log group signed in here; else None (blocking)."""
+    from views.removed_chat import _fetch
+
+    logged = avatar.get("logged") or {}
+    if not avatar.get("has_photo") or not logged.get("chat_id"):
+        return None
+    reader = media_share.reader_for(
+        account, logged["chat_id"], [avatar.get("master_tg_id"), *masters]
+    )
+    if reader is None:
+        return None
+    channel_id = peers.channel_id_from_marked(logged["chat_id"])
+    message_id = int(logged["message_id"])
+    copy = _fetch(reader, -channel_id, [message_id]).get(message_id)
+    found = _photo_of(copy)
+    return (found, reader, copy) if found is not None else None
+
+
+class Avatars:
+    def __init__(self, plugin, api):
+        self.plugin = plugin
+        self.api = api
+        self.handles = []
+        self._lock = threading.Lock()
+        self._recorded = {}  # (account, dialog) -> [(photo, original id, date)]
+        self._ids = set()  # ids of those photos: drawn with the trash icon
+        self._loading = set()
+        self._loaded_at = {}  # (account, dialog) -> monotonic time of the answer
+        self._told = {}  # (account, dialog) -> monotonic time the gallery was told
+        self._appending = False
+        self._updates = {}  # (account, dialog) -> args of its last dialogPhotosUpdate
+        self._icon = None
+        self._dates = {}  # logged photo id -> (set at, deleted at), ISO strings
+        self._watched = {}  # gallery hash -> its page listener (kept alive)
+        self._position = None  # ProfileGalleryView.getRealPosition
+        self._shown = None  # PhotoViewer.isVisible()
+
+    def install(self):
+        avatars = self
+
+        class AfterPhotos(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    avatars.photos_wanted(param.thisObject, param.args[0], param.getResult())
+                except Exception as exc:
+                    diag.error("avatars.photos", exc)
+
+        class AfterSetData(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    avatars.watch_gallery(param.thisObject)
+                except Exception as exc:
+                    diag.error("avatars.gallery", exc)
+
+        class AfterOverlays(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    avatars.update_profile(field(param.thisObject, "this$0"))
+                except Exception as exc:
+                    diag.error("avatars.profile", exc)
+
+        class AfterSwitch(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    avatars.viewer_dates(param.thisObject, int(param.args[0]))
+                except Exception as exc:
+                    diag.error("avatars.dates", exc)
+
+        class BeforeOpen(MethodHook):
+            def before_hooked_method(self, param):
+                try:
+                    if avatars.open_without_photo(param.thisObject):
+                        param.setResult(None)
+                except Exception as exc:
+                    diag.error("avatars.open", exc)
+
+        class AfterThumbs(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    avatars.draw_thumbs(param.thisObject, param.args[0])
+                except Exception as exc:
+                    diag.error("avatars.thumbs", exc)
+
+        class BeforeNotify(MethodHook):
+            def before_hooked_method(self, param):
+                try:
+                    avatars.photos_updated(param)
+                except Exception as exc:
+                    diag.error("avatars.update", exc)
+
+        self.handles += hook_methods(self.plugin, CONTROLLER, "getDialogPhotos", AfterPhotos())
+        self.handles += hook_methods(self.plugin, THUMBS, "onDraw", AfterThumbs())
+        self.handles += hook_methods(self.plugin, GALLERY, "setData", AfterSetData())
+        self.handles += hook_methods(
+            self.plugin, VIEWER, "setIsAboutToSwitchToIndex", AfterSwitch()
+        )
+        self.handles += hook_methods(
+            self.plugin, "org.telegram.ui.ProfileActivity", "openAvatar", BeforeOpen()
+        )
+        for name in ("onDraw", "setAlphaValue"):
+            self.handles += hook_methods(self.plugin, OVERLAYS, name, AfterOverlays())
+        self.handles += hook_methods(
+            self.plugin, NOTIFICATIONS, "postNotificationName", BeforeNotify()
+        )
+
+    def uninstall(self):
+        for handle in self.handles:
+            try:
+                self.plugin.unhook_method(handle)
+            except Exception:
+                pass
+        self.handles = []
+
+    # loading (background)
+
+    def photos_wanted(self, controller, dialog_id, holder):
+        """getDialogPhotos: the chat's logged photos are loaded once, then
+        put into its photos (again after the app reloaded them)."""
+        if self._appending or not self.api.configured or holder is None:
+            return
+        account = int(field(controller, "currentAccount") or 0)
+        key = (account, int(dialog_id))
+        with self._lock:
+            if key in self._recorded:
+                # Added (and the gallery told) once the app is done with it.
+                on_ui(lambda: self._append(*key))
+                if time.monotonic() - self._loaded_at.get(key, 0) < _FRESH_SECONDS:
+                    return
+            if key in self._loading:
+                return
+            self._loading.add(key)
+        in_background(lambda: self._load(*key))
+
+    def _load(self, account, dialog_id):
+        try:
+            chat_id = peers.marked_chat_id(account, dialog_id)
+            if chat_id is None:
+                return
+            try:
+                logged = self.api.avatars(peers.own_user_id(account), chat_id)
+            except ApiError as exc:
+                diag.log(f"avatars {chat_id}: {exc}")
+                return
+            recorded = []
+            masters = self.api.masters()
+            for avatar in logged:
+                found = logged_photo(account, avatar, masters)
+                if found is None:
+                    continue
+                photo, reader, copy = found
+                if not media_share.photo_sizes(reader, photo, copy):
+                    diag.log(f"avatars {chat_id}: logged photo unavailable")
+                    continue  # it can't be shown
+                original = avatar.get("photo_id")
+                recorded.append(
+                    (photo, int(original) if original else None, _epoch(avatar.get("date")))
+                )
+                self._dates[int(photo.id)] = (avatar.get("date"), avatar.get("deleted_at"))
+            with self._lock:
+                self._recorded[(account, dialog_id)] = recorded
+                self._loaded_at[(account, dialog_id)] = time.monotonic()
+                self._ids.update(int(p.id) for p, _o, _d in recorded)
+            if recorded:
+                diag.log(f"avatars {chat_id}: {len(recorded)} logged photos")
+                on_ui(lambda: self._append(account, dialog_id))
+        finally:
+            with self._lock:
+                self._loading.discard((account, dialog_id))
+
+    # the gallery (UI thread)
+
+    def _gone(self, photos, original, date):
+        """Whether a logged photo is not one the chat still has."""
+        for photo in photos:
+            if photo is None or int(photo.id) in self._ids:
+                continue
+            if original is not None and int(photo.id) == original:
+                return False
+            if original is None and abs(int(photo.date) - date) <= _SAME_PHOTO_SECONDS:
+                return False
+        return True
+
+    def _add_to(self, account, dialog_id, holder):
+        """Appends the logged photos the chat no longer has; whether any."""
+        photos = field(holder, "photos")
+        if photos is None:
+            return False
+        current = java_list(photos)
+        shown = {int(p.id) for p in current if p is not None}
+        added = False
+        for photo, original, date in self._recorded.get((account, dialog_id), []):
+            if int(photo.id) in shown or not self._gone(current, original, date):
+                continue
+            photos.add(photo)
+            shown.add(int(photo.id))
+            added = True
+        return added
+
+    def _viewing(self, dialog_id):
+        """Whether the photo viewer shows the chat's photos: its list is
+        built when it opens and a change meanwhile desyncs it from its
+        thumbnails (no swiping to the new ones)."""
+        from org.telegram.ui import PhotoViewer
+
+        if not PhotoViewer.hasInstance():
+            return False
+        viewer = PhotoViewer.getInstance()
+        if self._shown is None:
+            # The method (not the field, which stays set once it closed).
+            self._shown = java_class(VIEWER).getMethod("isVisible")
+        return bool(self._shown.invoke(viewer)) and int(
+            field(viewer, "avatarsDialogId") or 0
+        ) == int(dialog_id)
+
+    def _append(self, account, dialog_id, force=False):
+        """Puts the logged photos into the chat's photos and tells the
+        gallery, at most every few seconds (it reads the photos again)."""
+        from org.telegram.messenger import MessagesController, NotificationCenter
+
+        if not self.handles:
+            return  # unloaded
+        if not force and self._viewing(dialog_id):
+            on_ui(lambda: self._append(account, dialog_id), 1000)  # once it closed
+            return
+        self._appending = True  # getDialogPhotos calls back into photos_wanted
+        try:
+            holder = MessagesController.getInstance(account).getDialogPhotos(dialog_id)
+        finally:
+            self._appending = False
+        if holder is None:
+            return
+        key = (account, dialog_id)
+        added = self._add_to(account, dialog_id, holder)
+        if not added and time.monotonic() - self._told.get(key, 0) < _TELL_SECONDS:
+            return
+        self._told[key] = time.monotonic()
+        args = self._updates.get(key)
+        if args is None:
+            from java import jarray
+            from java.lang import Object
+
+            args = jarray(Object)([holder])
+        NotificationCenter.getInstance(account).postNotificationName(
+            NotificationCenter.dialogPhotosUpdate, args
+        )
+
+    def photos_updated(self, param):
+        """dialogPhotosUpdate: the logged photos are put back into the
+        (reloaded) list before the gallery redraws."""
+        from org.telegram.messenger import NotificationCenter
+
+        if int(param.args[0]) != int(NotificationCenter.dialogPhotosUpdate):
+            return
+        extra = param.args[1]
+        holder = next(
+            (a for a in (extra or []) if a is not None and hasattr(a, "photos")), None
+        )
+        if holder is None:
+            return
+        account = int(field(param.thisObject, "currentAccount") or 0)
+        dialog_id = int(field(holder, "dialogId") or 0)
+        self._updates[(account, dialog_id)] = extra
+        if (account, dialog_id) not in self._recorded:
+            return
+        if self._viewing(dialog_id):
+            on_ui(lambda: self._append(account, dialog_id), 1000)
+        else:
+            self._add_to(account, dialog_id, holder)
+
+    # the trash icon
+
+    def _logged_at(self, strip, index):
+        """Whether the strip's item `index` is a logged photo."""
+        for name in ("currentObjects", "currentPhotos"):
+            items = field(strip, name)
+            if items is None or not 0 <= index < items.size():
+                continue
+            item = items.get(index)
+            photo_id = getattr(item, "photoId", None) if name == "currentPhotos" else None
+            if photo_id is None:
+                photo_id = getattr(item, "id", None)
+            if photo_id is not None and int(photo_id) in self._ids:
+                return True
+        return False
+
+    def _alpha(self):
+        from config import ALPHA_CHOICES, DEFAULTS, DELETED_ALPHA
+
+        index = int(self.plugin.get_setting(DELETED_ALPHA, DEFAULTS[DELETED_ALPHA]) or 0)
+        return ALPHA_CHOICES[max(0, min(index, len(ALPHA_CHOICES) - 1))][1]
+
+    def draw_thumbs(self, strip, canvas):
+        """The photo viewer's thumbnails of logged photos: translucent, with a
+        small trash icon (GroupedPhotosListView: `imagesToDraw`, each
+        receiver's param its index)."""
+        if not self._ids:
+            return
+        receivers = field(strip, "imagesToDraw")
+        if receivers is None:
+            return
+        from org.telegram.messenger import AndroidUtilities
+
+        from android.graphics import Paint
+        from java import jint
+
+        icon, paint = self._trash()
+        # The strip draws its thumbnails opaque (it sets their alpha itself)
+        # over the viewer's black: black at the remaining opacity on top
+        # looks the same as the deleted message opacity.
+        shade = Paint()
+        shade.setColor(jint(int((1.0 - self._alpha()) * 255) << 24))
+        dp = AndroidUtilities.dp
+        size = dp(14)
+        for receiver in java_list(receivers):
+            if receiver is None or not self._logged_at(strip, int(receiver.getParam())):
+                continue
+            canvas.drawRect(
+                float(receiver.getImageX()),
+                float(receiver.getImageY()),
+                float(receiver.getImageX2()),
+                float(receiver.getImageY2()),
+                shade,
+            )
+            right = int(receiver.getImageX2()) - dp(2)
+            bottom = int(receiver.getImageY2()) - dp(2)
+            paint.setAlpha(0x99)
+            canvas.drawCircle(right - size / 2.0, bottom - size / 2.0, size * 0.8, paint)
+            icon.setAlpha(255)
+            icon.setBounds(right - size, bottom - size, right, bottom)
+            icon.draw(canvas)
+
+    # the profile's trash icon (under the menu button)
+
+    def watch_gallery(self, gallery):
+        """Redraws the icon as the profile's gallery changes page."""
+        key = int(gallery.hashCode())
+        if key in self._watched:
+            return
+        from java import dynamic_proxy, jclass
+
+        listener_class = jclass("androidx.viewpager.widget.ViewPager$OnPageChangeListener")
+        avatars = self
+
+        class Pages(dynamic_proxy(listener_class)):
+            def onPageScrolled(self, position, offset, pixels):
+                pass
+
+            def onPageSelected(self, position):
+                on_ui(lambda: avatars.update_profile(field(gallery, "this$0")))
+
+            def onPageScrollStateChanged(self, state):
+                pass
+
+        listener = Pages()
+        gallery.addOnPageChangeListener(listener)
+        self._watched[key] = listener
+
+    def _current_photo(self, gallery):
+        if self._position is None:
+            method = java_class(GALLERY).getDeclaredMethod("getRealPosition")
+            method.setAccessible(True)
+            self._position = method
+        photos = field(gallery, "photos")
+        index = int(self._position.invoke(gallery))
+        if photos is None or not 0 <= index < photos.size():
+            return None
+        return photos.get(index)
+
+    def update_profile(self, profile):
+        """Shows the icon while the expanded gallery shows a logged photo."""
+        if profile is None or not self._ids:
+            return
+        gallery = field(profile, "avatarsViewPager")
+        if gallery is None:
+            return
+        photo = self._current_photo(gallery)
+        shown = (
+            photo is not None
+            and int(photo.id) in self._ids
+            and bool(field(profile, "isPulledDown"))
+        )
+        mark = self._profile_mark(profile, create=shown)
+        if mark is None:
+            return
+        from android.view import View
+
+        mark.setVisibility(View.VISIBLE if shown else View.GONE)
+        mark.setAlpha(float(field(profile, "currentExpandAnimatorValue") or 1.0))
+
+    def _profile_mark(self, profile, create):
+        root = field(profile, "fragmentView")
+        if root is None or not hasattr(root, "addView"):
+            return None
+        for index in range(root.getChildCount()):
+            child = root.getChildAt(index)
+            if str(child.getTag()) == _MARK_TAG:
+                return child
+        if not create:
+            return None
+        from android.view import Gravity
+        from android.widget import FrameLayout, ImageView
+        from java import jint
+        from org.telegram.messenger import AndroidUtilities
+        from org.telegram.ui.ActionBar import ActionBar, Theme
+
+        from android.graphics import PorterDuff, PorterDuffColorFilter
+        from org.telegram.messenger import R
+
+        from hooks.render import _trash_drawable
+
+        dp = AndroidUtilities.dp
+        icon = _trash_drawable(R.drawable.msg_delete)
+        red = jint(Theme.getColor(Theme.key_text_RedBold))
+        icon.setColorFilter(PorterDuffColorFilter(red, PorterDuff.Mode.SRC_IN))
+        mark = ImageView(root.getContext())
+        mark.setTag(_MARK_TAG)
+        mark.setImageDrawable(icon)
+        mark.setPadding(dp(8), dp(8), dp(8), dp(8))
+        mark.setBackground(Theme.createCircleDrawable(dp(36), jint(0x66000000)))
+        params = FrameLayout.LayoutParams(dp(36), dp(36))
+        params.gravity = Gravity.TOP | Gravity.RIGHT
+        params.topMargin = AndroidUtilities.statusBarHeight + ActionBar.getCurrentActionBarHeight()
+        params.rightMargin = dp(10)
+        root.addView(mark, params)
+        return mark
+
+    # dates in the photo viewer
+
+    def viewer_dates(self, viewer, index):
+        """A logged photo's subtitle: when it was deleted and set (instead of
+        its upload to the log group)."""
+        photos = field(viewer, "avatarsArr")
+        if photos is None or not 0 <= index < photos.size():
+            return
+        photo = photos.get(index)
+        dates = self._dates.get(int(photo.id)) if photo is not None else None
+        container = field(viewer, "actionBarContainer")
+        if dates is None or container is None:
+            return
+        from views.history import _when
+
+        set_at, deleted_at = dates
+        parts = [f"deleted {_when(deleted_at)}" if deleted_at else "deleted"]
+        if set_at:
+            parts.append(f"set {_when(set_at)}")
+        text = " · ".join(parts)
+        container.setSubtitle(text[:1].upper() + text[1:])
+
+    # a chat without a photo
+
+    def open_without_photo(self, profile):
+        """openAvatar of a chat that has no photo but logged ones: the photo
+        viewer opens on the newest logged photo (with the rest)."""
+        from org.telegram.messenger import ImageLocation, MessagesController
+        from org.telegram.tgnet import TLRPC
+        from org.telegram.ui import PhotoViewer
+
+        account = int(profile.getCurrentAccount())
+        controller = MessagesController.getInstance(account)
+        user_id, chat_id = int(field(profile, "userId") or 0), int(field(profile, "chatId") or 0)
+        if user_id:
+            peer, dialog_id = controller.getUser(user_id), user_id
+            empty = TLRPC.TL_userProfilePhotoEmpty
+        else:
+            peer, dialog_id = controller.getChat(chat_id), -chat_id
+            empty = TLRPC.TL_chatPhotoEmpty
+        photo = getattr(peer, "photo", None) if peer is not None else None
+        if photo is not None and not isinstance(photo, empty):
+            return False  # the app opens it
+        recorded = self._recorded.get((account, dialog_id))
+        if not recorded:
+            controller.getDialogPhotos(dialog_id)  # loads the logged ones
+            return False
+        # The newest of the chat's photos (its own, then the logged ones).
+        holder = controller.getDialogPhotos(dialog_id)
+        if holder is not None:
+            self._add_to(account, dialog_id, holder)
+        listed = java_list(field(holder, "photos")) if holder is not None else []
+        first = next((p for p in listed if p is not None), recorded[0][0])
+        size = media_share.photo_size(first, 800)
+        if size is None:
+            return False
+        viewer = PhotoViewer.getInstance()
+        viewer.setParentActivity(profile)
+        viewer.openPhoto(
+            size.location, ImageLocation.getForPhoto(size, first), field(profile, "provider")
+        )
+        # The profile names no chat for a photo it doesn't show itself: the
+        # viewer is told, and gets the chat's photos (the logged ones).
+        from hook_utils import set_private_field
+        from java import jlong
+
+        set_private_field(viewer, "avatarsDialogId", jlong(dialog_id))
+        self._told.pop((account, dialog_id), None)
+        self._append(account, dialog_id, force=True)
+
+        return True
+
+    def _trash(self):
+        if self._icon is None:
+            from android.graphics import Paint, PorterDuff, PorterDuffColorFilter
+            from java import jint
+            from org.telegram.messenger import R
+            from org.telegram.ui.ActionBar import Theme
+
+            from hooks.render import _trash_drawable
+
+            icon = _trash_drawable(R.drawable.msg_delete)
+            red = jint(Theme.getColor(Theme.key_text_RedBold))
+            icon.setColorFilter(PorterDuffColorFilter(red, PorterDuff.Mode.SRC_IN))
+            paint = Paint(Paint.ANTI_ALIAS_FLAG)
+            paint.setColor(jint(0x99000000 - (1 << 32)))  # ARGB as a signed int
+            self._icon = (icon, paint)
+        return self._icon

src/hooks/chat.py

@@ -27,6 +27,7 @@ from base_plugin import MethodHook
 
 import diag
 import header
+import media_share
 import peers
 import store
 import tg
@@ -167,6 +168,75 @@ def _rich_plain_text(rich):
     return "\n".join(_rich_text(b.text) for b in blocks)
 
 
+def build_from_copy(account, dialog_id, message_id, state, copy, strip):
+    """A deleted message rebuilt from its logged copy (`strip`: the copy
+    starts with the logger's header) with the server's `state`."""
+    from java.util import ArrayList
+    from org.telegram.messenger import MessageObject, MessagesController
+    from org.telegram.tgnet import TLRPC
+
+    controller = MessagesController.getInstance(account)
+    msg = TLRPC.TL_message()
+    msg.id = message_id
+    msg.dialog_id = dialog_id
+    msg.peer_id = controller.getPeer(dialog_id)
+    sender = state.get("sender_id")
+    msg.from_id = (
+        controller.getPeer(_android_peer_id(sender)) if sender else msg.peer_id
+    )
+    msg.flags |= 256
+    msg.date = _epoch(state.get("source_date")) or int(copy.date)
+    if int(state.get("edit_count") or 0) > 0:
+        msg.edit_date = _epoch(state.get("edit_date")) or msg.date
+        msg.flags |= 32768  # edited: the app draws its pencil
+    text = copy.message or ""
+    entities = [(int(e.offset), int(e.length), e) for e in java_list(copy.entities)]
+    if strip:
+        text, entities = header.strip(text, entities)
+    msg.message = text
+    msg.entities = ArrayList()
+    for offset, length, entity in entities:
+        entity.offset = offset
+        entity.length = length
+        msg.entities.add(entity)
+    media = getattr(copy, "media", None)
+    if media is not None and not isinstance(media, TLRPC.TL_messageMediaEmpty):
+        msg.media = media
+        msg.flags |= 512
+    rich = getattr(copy, "rich_message", None)
+    if rich is not None:
+        if strip:
+            _strip_rich_header(rich)
+        plain = _rich_plain_text(rich)
+        if plain:  # never a blank bubble: else it stays rich
+            # Text only: a normal bubble, as the original was.
+            msg.message = plain
+            msg.entities = ArrayList()
+        else:
+            msg.rich_message = rich
+            msg.flags2 |= 1 << 12
+    own = peers.own_user_id(account)
+    msg.out = sender is not None and int(sender) == own
+    msg.unread = False
+    if sender:
+        _remember_sender(account, _android_peer_id(sender))
+    return MessageObject(account, msg, True, True)
+
+
+def _remember_sender(account, peer_id):
+    """Snapshot the sender's profile while the app can still read it, so it
+    can be shown later when the app no longer has it."""
+    try:
+        import people
+
+        if peer_id > 0:
+            people.user(account, peer_id)
+        elif peer_id < 0:
+            people.chat(account, -peer_id)
+    except Exception as exc:
+        diag.error("chat.remember_sender", exc)
+
+
 class ChatSync:
     def __init__(self, plugin, api):
         self.plugin = plugin
@@ -176,6 +246,8 @@ class ChatSync:
         self._inflight = set()  # (account, dialog) with a sync running
         self._exists = {}  # (account, dialog) -> ids Telegram still has
         self._restoring = set()  # (account, dialog, id) in flight
+        self._copies = {}  # (account, dialog, id) -> (copy, strip) read, not placed
+        self._placing = False  # a restored message is being placed
         self._fragments = {}  # (account, dialog) -> last ChatActivity seen
 
     def install(self):
@@ -194,6 +266,18 @@ class ChatSync:
             self.plugin, CHAT, "updateMessagesVisiblePart", AfterVisible()
         )
 
+        class BeforeBadge(MethodHook):
+            def before_hooked_method(self, param):
+                if sync._placing:
+                    param.setResult(None)  # a restored message is no arrival
+
+        self.handles += hook_methods(
+            self.plugin,
+            "org.telegram.ui.Components.chat.layouts.ChatActivitySideControlsButtonsLayout",
+            "setButtonCount",
+            BeforeBadge(),
+        )
+
     def uninstall(self):
         for name in EDIT_UPDATES:
             try:
@@ -227,6 +311,8 @@ class ChatSync:
         if chat_id is None:
             return
         key = (account, dialog_id)
+        if store.is_removed_dialog(account, dialog_id):
+            return  # its history brings its state (hooks/removed_chats.py)
         self._fragments[key] = fragment
         if key in self._inflight:
             return
@@ -247,9 +333,10 @@ class ChatSync:
             # cached answers (they were never in the app's own database).
             if self.plugin.get_setting(RESTORE_MISSING, True):
                 present = set(ids)
+                revealed = store.is_revealed(account, dialog_id)
                 gone = [
                     s
-                    for s in store.cached_deleted(account, dialog_id, low, high)
+                    for s in store.cached_deleted(account, dialog_id, low, high, revealed)
                     if int(s["message_id"]) not in present
                 ]
                 if gone:
@@ -355,13 +442,22 @@ class ChatSync:
     # restoring (background -> UI)
 
     def _restore(self, fragment, account, dialog_id, chat_id, me, states):
+        revealed = store.is_revealed(account, dialog_id)
         for state in states:
             message_id = int(state["message_id"])
-            if store.is_hidden(account, dialog_id, message_id):
+            if not revealed and store.is_hidden(account, dialog_id, message_id):
-                continue  # the user removed it from view
+                continue  # removed from view
             key = (account, dialog_id, message_id)
             if key in self._restoring:
                 continue
+            fetched = self._copies.get(key)
+            if fetched is not None:  # read before; not placed yet
+                on_ui(
+                    lambda f=fetched, m=message_id, st=state: self._insert(
+                        fragment, account, dialog_id, m, st, *f
+                    )
+                )
+                continue
             pointer, strip = self._source(me, chat_id, message_id, state)
             if pointer is None:
                 continue
@@ -383,8 +479,24 @@ class ChatSync:
         return state.get("logged"), True
 
     def _fetch_copy(self, fragment, account, dialog_id, message_id, state, pointer, strip):
+        """Reads the copy through a member of its log group signed in here
+        (this account, else the master); another account's download of its
+        media lets this one show it (media_share)."""
         channel_id = peers.channel_id_from_marked(pointer["chat_id"])
         copy_id = int(pointer["message_id"])
+        reader = media_share.reader_for(account, pointer["chat_id"], self.api.masters())
+        if reader is None:
+            self._restoring.discard((account, dialog_id, message_id))
+            diag.log(f"copy {pointer}: no account here is in its log group")
+            return
+
+        def insert(copy):
+            if reader != account:
+                ready, total = media_share.copy_media(reader, copy)
+                if total:
+                    diag.log(f"copy {pointer}: {ready}/{total} media through account {reader}")
+            self._copies[(account, dialog_id, message_id)] = (copy, strip)
+            on_ui(lambda: self._insert(fragment, account, dialog_id, message_id, state, copy, strip))
 
         def done(found, error):
             copy = (found or {}).get(copy_id)
@@ -392,80 +504,55 @@ class ChatSync:
                 self._restoring.discard((account, dialog_id, message_id))
                 diag.log(f"copy {pointer} unavailable: {error}")
                 return
-            on_ui(lambda: self._insert(fragment, account, dialog_id, message_id, state, copy, strip))
+            in_background(lambda: insert(copy))
 
-        tg.get_messages(account, -channel_id, [copy_id], done)
+        tg.get_messages(reader, -channel_id, [copy_id], done)
-
-    def _build(self, account, dialog_id, message_id, state, copy, strip):
-        from java.util import ArrayList
-        from org.telegram.messenger import MessageObject, MessagesController
-        from org.telegram.tgnet import TLRPC
-
-        controller = MessagesController.getInstance(account)
-        msg = TLRPC.TL_message()
-        msg.id = message_id
-        msg.dialog_id = dialog_id
-        msg.peer_id = controller.getPeer(dialog_id)
-        sender = state.get("sender_id")
-        msg.from_id = (
-            controller.getPeer(_android_peer_id(sender)) if sender else msg.peer_id
-        )
-        msg.flags |= 256
-        msg.date = _epoch(state.get("source_date")) or int(copy.date)
-        if int(state.get("edit_count") or 0) > 0:
-            msg.edit_date = _epoch(state.get("edit_date")) or msg.date
-            msg.flags |= 32768  # edited: the app draws its pencil
-        text = copy.message or ""
-        entities = [(int(e.offset), int(e.length), e) for e in java_list(copy.entities)]
-        if strip:
-            text, entities = header.strip(text, entities)
-        msg.message = text
-        msg.entities = ArrayList()
-        for offset, length, entity in entities:
-            entity.offset = offset
-            entity.length = length
-            msg.entities.add(entity)
-        media = getattr(copy, "media", None)
-        if media is not None and not isinstance(media, TLRPC.TL_messageMediaEmpty):
-            msg.media = media
-            msg.flags |= 512
-        rich = getattr(copy, "rich_message", None)
-        if rich is not None:
-            if strip:
-                _strip_rich_header(rich)
-            plain = _rich_plain_text(rich)
-            if plain:  # never a blank bubble: else it stays rich
-                # Text only: a normal bubble, as the original was.
-                msg.message = plain
-                msg.entities = ArrayList()
-            else:
-                msg.rich_message = rich
-                msg.flags2 |= 1 << 12
-        own = peers.own_user_id(account)
-        msg.out = sender is not None and int(sender) == own
-        msg.unread = False
-        return MessageObject(account, msg, True, True)
 
     def _insert(self, fragment, account, dialog_id, message_id, state, copy, strip):
         """Hands the rebuilt message to the chat the way new messages arrive
         (ChatActivity.processNewMessages), so the app places it and keeps its
         own dicts and adapter consistent."""
-        self._restoring.discard((account, dialog_id, message_id))
+        key = (account, dialog_id, message_id)
+        self._restoring.discard(key)
         messages = field(fragment, "messages")
         if messages is None:
             return
         if _loaded(fragment, message_id) or any(
             m is not None and int(m.getId()) == message_id for m in java_list(messages)
         ):
+            self._copies.pop(key, None)
             return
+        ids = [int(m.getId()) for m in java_list(messages) if m is not None and int(m.getId()) > 0]
+        at_end = _end_reached(fragment, "forwardEndReached")
+        if not at_end and (not ids or not min(ids) < message_id < max(ids)):
+            return  # outside what is loaded: placed when that part is
+        self._copies.pop(key, None)
         from java import jboolean
         from java.util import ArrayList
 
-        obj = self._build(account, dialog_id, message_id, state, copy, strip)
+        obj = build_from_copy(account, dialog_id, message_id, state, copy, strip)
         store.mark_deleted(account, dialog_id, [message_id])
         batch = ArrayList()
         batch.add(obj)
-        _process_new_messages().invoke(fragment, batch, jboolean(False))
+        # Away from the newest messages the app only counts arrivals (the
+        # scroll-down badge); with the end "loaded" it places the message
+        # by its id among the loaded ones.
+        # It is no arrival: the unread count (scroll-down badge) stays.
+        flags = field(fragment, "forwardEndReached")
+        unread = int(field(fragment, "newUnreadMessageCount") or 0)
+        if not at_end:
+            flags[0] = True
+        self._placing = True
+        try:
+            _process_new_messages().invoke(fragment, batch, jboolean(False))
+        finally:
+            self._placing = False
+            if not at_end:
+                flags[0] = False
+            from hook_utils import set_private_field
+            from java import jint
+
+            set_private_field(fragment, "newUnreadMessageCount", jint(unread))
         diag.log(f"restored {dialog_id}/{message_id}")
 
     # cells (UI thread)
@@ -552,6 +639,35 @@ class ChatSync:
             on_ui(redraw)
         return removed
 
+    def restore_hidden(self, fragment):
+        """The chat reveals its hidden messages: those deleted in the loaded
+        range come back from the logger (UI thread). Returns how many."""
+        account = int(fragment.getCurrentAccount())
+        dialog_id = int(fragment.getDialogId())
+        chat_id = peers.marked_chat_id(account, dialog_id)
+        ids = [
+            int(m.getId())
+            for m in java_list(field(fragment, "messages"))
+            if m is not None and int(m.getId()) > 0
+        ]
+        if chat_id is None or not ids:
+            return 0
+        low = 1 if _end_reached(fragment, "endReached") else min(ids)
+        high = _MAX_ID if _end_reached(fragment, "forwardEndReached") else max(ids)
+        present = set(ids)
+        hidden = set(store.hidden_deleted(account, dialog_id))
+        gone = [
+            s
+            for s in store.cached_deleted(account, dialog_id, low, high, True)
+            if int(s["message_id"]) in hidden and int(s["message_id"]) not in present
+        ]
+        if gone:
+            me = peers.own_user_id(account)
+            in_background(
+                lambda: self._restore(fragment, account, dialog_id, chat_id, me, gone)
+            )
+        return len(gone)
+
     def drop_from_view(self, account, dialog_id, ids):
         """Takes messages out of the open chat's list (UI thread)."""
         fragment = self._open_chat(account, dialog_id)

src/hooks/delete_dialog.py

@@ -5,10 +5,10 @@
   filtered out of MessagesStorage.markMessagesAsDeleted and the
   messagesDeleted notification; the server's echo then counts as someone
   else's deletion, see hooks/deletes.py). Unticked (default), it is deleted
-  as usual and remembered as hidden, so it is never restored from the
+  as usual and hidden (on the server, for every client), so it is not
-  logger here;
+  restored from the logger;
 - already deleted (kept) messages get a plain dialog without checkboxes
-  whose button removes them from view (locally only) and hides them.
+  whose button removes them from this device's view and hides them.
 
 AlertsCreator.createDeleteMessagesAlert(fragment, user, chat, encryptedChat,
 chatInfo, mergeDialogId, selectedMessage, selectedMessages[2], group,
@@ -53,9 +53,10 @@ def _deleted(message):
 
 
 class DeleteDialog:
-    def __init__(self, plugin, chats):
+    def __init__(self, plugin, chats, hidden):
         self.plugin = plugin
         self.chats = chats  # hooks/chat.ChatSync
+        self.hidden = hidden  # hooks/hidden.Hidden
         self.handles = []
         self._choice = None  # [checked] of the open dialog
         self._keeping = {}  # (account, id) -> monotonic time to keep until
@@ -169,12 +170,14 @@ class DeleteDialog:
 
         count = len(targets)
         builder = AlertDialogBuilder(fragment.getParentActivity())
-        builder.set_title("Remove deleted message" + ("s" if count > 1 else ""))
+        builder.set_title("Hide deleted message" + ("s" if count > 1 else ""))
         builder.set_message(
             ("These messages were" if count > 1 else "This message was")
-            + " already deleted. Remove "
+            + " already deleted. Hide "
             + ("them" if count > 1 else "it")
-            + " from view on this device?"
+            + " from this chat? The logger keeps "
+            + ("them" if count > 1 else "it")
+            + ": the chat menu reveals hidden messages."
         )
 
         def remove(_builder, _which):
@@ -188,14 +191,15 @@ class DeleteDialog:
             if hide_dim is not None:
                 hide_dim.run()
 
-        builder.set_positive_button("Remove", remove)
+        builder.set_positive_button("Hide", remove)
         builder.set_negative_button("Back")
         builder.make_button_red(AlertDialogBuilder.BUTTON_POSITIVE)
         builder.set_on_dismiss_listener(dismissed)
         builder.show()
 
     def _remove_from_view(self, targets):
-        """Removes already deleted messages from this device only."""
+        """Hides already deleted messages (on the server) and takes them out
+        of this device's chat."""
         from java import jboolean, jint, jlong
         from java.lang import Object
         from java.util import ArrayList
@@ -207,8 +211,9 @@ class DeleteDialog:
             account, dialog_id, message_id = _key(message)
             by_chat.setdefault((account, dialog_id), []).append(message_id)
         for (account, dialog_id), ids in by_chat.items():
-            store.hide(account, dialog_id, ids)
+            # Its cached state stays: revealing the chat's hidden messages
-            store.forget_deleted(account, dialog_id, ids)
+            # restores it from there.
+            self.hidden.hide(account, dialog_id, ids)
             channel_id = 0
             if dialog_id < 0:
                 chat = MessagesController.getInstance(account).getChat(-dialog_id)
@@ -249,7 +254,7 @@ class DeleteDialog:
             store.mark_deleted(account, dialog_id, ids)
             diag.log(f"keeping own deletion {dialog_id}: {ids}")
             return list(ids)
-        store.hide(account, dialog_id, ids)
+        self.hidden.hide(account, dialog_id, ids)
         return []
 
     def filter_storage(self, param):

src/hooks/hidden.py

@@ -0,0 +1,202 @@
+"""Deleted messages removed from view ("hidden"), marked on the server so
+every client skips them; a chat can reveal its hidden messages again.
+
+- `flush` sends the marks waiting in the store to the server;
+- the chat menu toggles revealing (per chat, until the app restarts): the
+  hidden deleted messages come back from the logger, then go again;
+- a revealed hidden message has "Unhide" in its menu (hooks/menu.py), and
+  while revealing, the selection bar gets an Unhide button for those
+  selected (ChatActivity.createActionMode adds the bar's items,
+  updateActionModeTitle runs on every selection change; probed on 12.10.6).
+"""
+
+import threading
+
+from base_plugin import MethodHook
+
+import diag
+import peers
+import store
+from api import ApiError
+from jutil import field, hook_methods, in_background, on_ui
+
+CHAT = "org.telegram.ui.ChatActivity"
+_UNHIDE_ID = 0x4D4C01  # the selection bar item
+
+
+def _bulletin(text, error=False):
+    def show():
+        from ui.bulletin import BulletinHelper
+
+        (BulletinHelper.show_error if error else BulletinHelper.show_info)(text)
+
+    on_ui(show)
+
+
+def _selected(chat):
+    """MessageObjects selected in the chat (both halves of a merged chat)."""
+    out = []
+    for array in field(chat, "selectedMessagesIds") or []:
+        if array is None:
+            continue
+        for index in range(array.size()):
+            message = array.valueAt(index)
+            if message is not None:
+                out.append(message)
+    return out
+
+
+def is_revealed_hidden(message):
+    """A hidden deleted message shown because its chat reveals them."""
+    if message is None or int(message.getId()) <= 0:
+        return False
+    account, dialog_id = int(message.currentAccount), int(message.getDialogId())
+    message_id = int(message.getId())
+    return (
+        store.is_revealed(account, dialog_id)
+        and store.is_deleted(account, dialog_id, message_id)
+        and store.is_hidden(account, dialog_id, message_id)
+    )
+
+
+class Hidden:
+    def __init__(self, plugin, api, chats):
+        self.plugin = plugin
+        self.api = api
+        self.chats = chats  # hooks/chat.ChatSync
+        self.handles = []
+        self._flushing = threading.Lock()
+        self._items = {}  # ChatActivity classGuid -> the bar's Unhide item
+
+    def install(self):
+        hidden = self
+
+        class AfterActionMode(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    hidden.add_bar_item(param.thisObject)
+                except Exception as exc:
+                    diag.error("hidden.bar", exc)
+
+        class AfterTitle(MethodHook):
+            def after_hooked_method(self, param):
+                try:
+                    hidden.update_bar_item(param.thisObject)
+                except Exception as exc:
+                    diag.error("hidden.title", exc)
+
+        self.handles += hook_methods(self.plugin, CHAT, "createActionMode", AfterActionMode())
+        self.handles += hook_methods(self.plugin, CHAT, "updateActionModeTitle", AfterTitle())
+        in_background(self.flush)
+
+    def uninstall(self):
+        for handle in self.handles:
+            try:
+                self.plugin.unhook_method(handle)
+            except Exception:
+                pass
+        self.handles = []
+
+    # the server
+
+    def flush(self):
+        """Sends hidden marks waiting in the store (background)."""
+        if not self.api.configured or not self._flushing.acquire(blocking=False):
+            return
+        try:
+            for (account, dialog_id, hidden), ids in store.pending_hidden().items():
+                me = peers.own_user_id(account)
+                chat_id = peers.marked_chat_id(account, dialog_id)
+                if not me or chat_id is None:
+                    continue
+                try:
+                    self.api.hidden(me, chat_id, ids, hidden)
+                except ApiError as exc:
+                    diag.log(f"hidden {chat_id}: {exc}")
+                    continue
+                store.hidden_sent(account, dialog_id, ids, hidden)
+                diag.log(f"{'hid' if hidden else 'unhid'} {chat_id}: {ids}")
+        finally:
+            self._flushing.release()
+
+    def hide(self, account, dialog_id, ids, hidden=True):
+        store.set_hidden(account, dialog_id, ids, hidden)
+        in_background(self.flush)
+
+    # revealing (chat menu)
+
+    def toggle_reveal(self, fragment):
+        account = int(fragment.getCurrentAccount())
+        dialog_id = int(fragment.getDialogId())
+        on = not store.is_revealed(account, dialog_id)
+        store.reveal(account, dialog_id, on)
+        if on:
+            count = self.chats.restore_hidden(fragment)
+            # And the server's answer, for those not cached.
+            store.expire_coverage(account, dialog_id)
+            self.chats.schedule(fragment)
+            _bulletin(
+                f"Message Logger: revealing hidden deleted messages ({count} loaded)"
+                if count
+                else "Message Logger: revealing hidden deleted messages"
+            )
+        else:
+            ids = store.hidden_deleted(account, dialog_id)
+            self.chats.drop_from_view(account, dialog_id, ids)
+            _bulletin("Message Logger: hidden deleted messages are hidden again")
+
+    # unhiding
+
+    def unhide(self, messages):
+        by_chat = {}
+        for message in messages:
+            if is_revealed_hidden(message):
+                key = (int(message.currentAccount), int(message.getDialogId()))
+                by_chat.setdefault(key, []).append(int(message.getId()))
+        for (account, dialog_id), ids in by_chat.items():
+            self.hide(account, dialog_id, ids, hidden=False)
+            self.chats.refresh_dialog(account, dialog_id, ids)
+        count = sum(len(ids) for ids in by_chat.values())
+        if count:
+            _bulletin(f"Message Logger: {count} message{'s' if count > 1 else ''} unhidden")
+
+    # the selection bar
+
+    def add_bar_item(self, chat):
+        """The bar's Unhide item (also for a chat opened before the plugin
+        loaded: its bar exists already)."""
+        bar = chat.getActionBar()
+        menu = bar.createActionMode() if bar is not None else None
+        if menu is None:
+            return None
+        from android.view import View
+        from android_utils import OnClickListener
+        from org.telegram.messenger import AndroidUtilities, R
+
+        item = menu.getItem(_UNHIDE_ID)
+        if item is None:
+            item = menu.addItemWithWidth(
+                _UNHIDE_ID, R.drawable.msg_views, AndroidUtilities.dp(54)
+            )
+            item.setContentDescription("Unhide")
+            item.setVisibility(View.GONE)
+
+        def chosen(_view):
+            self.unhide(_selected(chat))
+            chat.clearSelectionMode()
+
+        item.setOnClickListener(OnClickListener(chosen))
+        self._items[int(chat.getClassGuid())] = item
+        return item
+
+    def update_bar_item(self, chat):
+        item = self._items.get(int(chat.getClassGuid())) or self.add_bar_item(chat)
+        if item is None:
+            return
+        from android.view import View
+
+        account, dialog_id = int(chat.getCurrentAccount()), int(chat.getDialogId())
+        show = store.is_revealed(account, dialog_id) and any(
+            is_revealed_hidden(m) for m in _selected(chat)
+        )
+        item.setVisibility(View.VISIBLE if show else View.GONE)

src/hooks/menu.py

@@ -1,4 +1,5 @@
-""""Edit history" in the message menu itself, right above Forward.
+""""Edit history" (and "Unhide" for a revealed hidden message, see
+hooks/hidden.py) in the message menu itself, right above Forward.
 
 Plugin menu items (MenuItemData) only go into the menu's Plugins folder, and
 exteraGram drops option ids it doesn't know from fillMessageMenu's lists, so
@@ -12,10 +13,12 @@ from base_plugin import MethodHook
 
 import diag
 import store
+from hooks.hidden import is_revealed_hidden
 from jutil import field, hook_methods, java_class
 
 CHAT = "org.telegram.ui.ChatActivity"
-_LABEL = "Edit history"
+_HISTORY = "Edit history"
+_UNHIDE = "Unhide"
 
 
 def _worth_it(message):
@@ -58,9 +61,10 @@ def _find_item(view, text):
 
 
 class Menu:
-    def __init__(self, plugin, open_history):
+    def __init__(self, plugin, open_history, unhide):
         self.plugin = plugin
         self.open_history = open_history
+        self.unhide = unhide  # hooks/hidden.Hidden.unhide
         self.handles = []
         self._close = None
 
@@ -87,39 +91,49 @@ class Menu:
     def add_item(self, chat):
         message = field(chat, "selectedObject")
         popup = field(chat, "scrimPopupWindow")
-        if popup is None or not _worth_it(message):
+        if popup is None or message is None:
             return
-        # The createMenu overloads call one another: add the row once.
+        from org.telegram.messenger import R
-        if _find_item(popup.getContentView(), _LABEL) is not None:
-            return
-        from org.telegram.messenger import LocaleController, R
 
+        rows = []
+        if _worth_it(message):
+            rows.append((_HISTORY, R.drawable.msg_edit, lambda: self.open_history(message)))
+        if is_revealed_hidden(message):
+            rows.append((_UNHIDE, R.drawable.msg_views, lambda: self.unhide([message])))
+        # The createMenu overloads call one another: add the rows once.
         content = popup.getContentView()
+        if not rows or _find_item(content, rows[0][0]) is not None:
+            return
+        from org.telegram.messenger import LocaleController
+
         # Forward is a row in some chats and an icon in the bottom action bar
-        # in others (groups): then the row goes first in the list.
+        # in others (groups): then the rows go first in the list.
         anchor = _find_item(
             content, str(LocaleController.getString(R.string.Forward))
         ) or _first_item(content)
         if anchor is None:
-            diag.log("menu: no row to place Edit history by")
+            diag.log("menu: no row to place the rows by")
             return
-        forward = anchor
+        for label, icon, action in rows:
+            self._add_row(chat, anchor, label, icon, action)
+
+    def _add_row(self, chat, anchor, label, icon, action):
         from android_utils import OnClickListener
         from org.telegram.ui.ActionBar import ActionBarMenuSubItem
 
-        parent = forward.getParent()
+        parent = anchor.getParent()
-        item = ActionBarMenuSubItem(forward.getContext(), False, False)
+        item = ActionBarMenuSubItem(anchor.getContext(), False, False)
-        item.setTextAndIcon(_LABEL, R.drawable.msg_edit)
+        item.setTextAndIcon(label, icon)
-        color = forward.getTextView().getCurrentTextColor()
+        color = anchor.getTextView().getCurrentTextColor()
         item.setColors(color, color)
-        item.setMinimumWidth(forward.getMinimumWidth())
+        item.setMinimumWidth(anchor.getMinimumWidth())
 
         def chosen(_view):
             self._close_menu(chat)
-            self.open_history(message)
+            action()
 
         item.setOnClickListener(OnClickListener(chosen))
-        parent.addView(item, parent.indexOfChild(forward), forward.getLayoutParams())
+        parent.addView(item, parent.indexOfChild(anchor), anchor.getLayoutParams())
 
     def _close_menu(self, chat):
         if self._close is None:

src/jutil.py

@@ -36,15 +36,76 @@ def methods_named(class_name, name):
     return [m for m in cls.getDeclaredMethods() if m.getName() == name]
 
 
+# Hook profiling: per hook name, [calls, total ns, max ns]. Off in releases.
+PROFILE = {}
+_PROFILE_ON = False
+
+
+def _profiled(hook, label):
+    """Wrap a MethodHook so each call's time is counted under `label`."""
+    if not _PROFILE_ON:
+        return hook
+    import time as _time
+
+    from base_plugin import MethodHook
+
+    inner = hook
+
+    class Timed(MethodHook):
+        def before_hooked_method(self, param):
+            start = _time.perf_counter_ns()
+            try:
+                if hasattr(inner, "before_hooked_method"):
+                    inner.before_hooked_method(param)
+            finally:
+                _account(label + ":before", _time.perf_counter_ns() - start)
+
+        def after_hooked_method(self, param):
+            start = _time.perf_counter_ns()
+            try:
+                if hasattr(inner, "after_hooked_method"):
+                    inner.after_hooked_method(param)
+            finally:
+                _account(label + ":after", _time.perf_counter_ns() - start)
+
+    return Timed()
+
+
+def _account(label, spent):
+    entry = PROFILE.get(label)
+    if entry is None:
+        entry = PROFILE[label] = [0, 0, 0]
+    entry[0] += 1
+    entry[1] += spent
+    if spent > entry[2]:
+        entry[2] = spent
+    if spent > 16_000_000:
+        diag.log(f"SLOW {label}: {spent / 1e6:.1f}ms")
+
+
+def profile_report(reset=True):
+    rows = sorted(PROFILE.items(), key=lambda kv: -kv[1][1])
+    lines = [
+        f"{label}: n={n} total={total / 1e6:.1f}ms max={peak / 1e6:.1f}ms"
+        for label, (n, total, peak) in rows
+        if n
+    ]
+    if reset:
+        PROFILE.clear()
+    return lines
+
+
 def hook_methods(plugin, class_name, name, hook, arity=None):
     """Hooks every overload of `class_name.name` (or those with `arity`
     parameters); returns the unhook handles."""
     handles = []
+    short = class_name.rsplit(".", 1)[-1]
     for method in methods_named(class_name, name):
         if arity is not None and len(method.getParameterTypes()) != arity:
             continue
         try:
-            handle = plugin.hook_method(method, hook)
+            label = f"{short}.{name}/{len(method.getParameterTypes())}"
+            handle = plugin.hook_method(method, _profiled(hook, label))
             if handle is not None:
                 handles.append(handle)
         except Exception as exc:
@@ -67,6 +128,41 @@ def field(obj, name):
         return None
 
 
+_FIELDS = {}
+_MISSING = object()
+
+
+def cached_field(obj, name):
+    """`field()` for hot paths: the reflected Field is resolved once per
+    (class, name) instead of on every call (per-frame draw hooks)."""
+    try:
+        cls = obj.getClass()
+        key = (cls.getName(), name)
+    except Exception:
+        return field(obj, name)
+    found = _FIELDS.get(key, _MISSING)
+    if found is _MISSING:
+        found, current = None, cls
+        while current is not None:
+            try:
+                found = current.getDeclaredField(name)
+                found.setAccessible(True)
+                break
+            except Exception:
+                found = None
+                try:
+                    current = current.getSuperclass()
+                except Exception:
+                    current = None
+        _FIELDS[key] = found
+    if found is None:
+        return field(obj, name)
+    try:
+        return found.get(obj)
+    except Exception:
+        return field(obj, name)
+
+
 def on_ui(fn, delay_ms=0):
     from android_utils import run_on_ui_thread
 

src/main.py

@@ -15,13 +15,17 @@ from config import (
     REPORT_DELETIONS,
     RESTORE_MISSING,
     SERVER_URL,
+    SHOW_REMOVED_CHATS,
 )
+from hooks.avatars import Avatars
 from hooks.chat import EDIT_UPDATES, ChatSync
 from hooks.comments import CommentsButton
 from hooks.delete_dialog import DeleteDialog
 from hooks.deletes import UPDATES, Deletes
+from hooks.hidden import Hidden
 from hooks.links import Links
 from hooks.menu import Menu
+from hooks.removed_chats import RemovedChats
 from hooks.render import Render
 from jutil import in_background, on_ui
 from views.comments import Comments
@@ -43,19 +47,25 @@ class MessageLoggerPlugin(BasePlugin):
         self.history = History(self, self.api)
         self.render = Render(self)
         self.chats = ChatSync(self, self.api)
-        self.delete_dialog = DeleteDialog(self, self.chats)
+        self.hidden = Hidden(self, self.api, self.chats)
+        self.delete_dialog = DeleteDialog(self, self.chats, self.hidden)
         self.deletes = Deletes(self, self.api, self.chats, self.delete_dialog)
         self.links = Links(self)
-        self.menu = Menu(self, self.history.open)
+        self.menu = Menu(self, self.history.open, self.hidden.unhide)
         self.comments = CommentsButton(self, Comments(self, self.api).open)
+        self.avatars = Avatars(self, self.api)
+        self.removed_chats = RemovedChats(self, self.api)
         parts = (
             self.render,
             self.chats,
+            self.hidden,
             self.deletes,
             self.delete_dialog,
             self.links,
             self.menu,
             self.comments,
+            self.avatars,
+            self.removed_chats,
         )
         for part in parts:
             try:
@@ -65,14 +75,24 @@ class MessageLoggerPlugin(BasePlugin):
         self.add_menu_item(
             MenuItemData(
                 menu_type=MenuItemType.CHAT_ACTION_MENU,
+                item_id="message_logger_reset",
                 text="Reset deleted/edited marks",
                 icon="msg_clear",
                 on_click=self._reset_chat,
             )
         )
+        self.add_menu_item(
+            MenuItemData(
+                menu_type=MenuItemType.CHAT_ACTION_MENU,
+                item_id="message_logger_reveal",
+                text="Reveal hidden messages",
+                icon="msg_views",
+                on_click=self._toggle_reveal,
+            )
+        )
         in_background(accounts.refresh)
         in_background(self.purge_cache)
-        diag.log("loaded")
+        diag.log(f"loaded (server {self.get_setting(SERVER_URL, '') or 'unset'})")
 
     def _reset_chat(self, context):
         """Chat menu: forget this chat's cached marks and check it again."""
@@ -89,6 +109,19 @@ class MessageLoggerPlugin(BasePlugin):
         except Exception as exc:
             diag.error("reset chat", exc)
 
+    def _toggle_reveal(self, context):
+        """Chat menu: show the chat's hidden deleted messages (or hide them
+        again)."""
+        try:
+            fragment = context.get("fragment")
+            if fragment is None:
+                from client_utils import get_last_fragment
+
+                fragment = get_last_fragment()
+            self.hidden.toggle_reveal(fragment)
+        except Exception as exc:
+            diag.error("reveal", exc)
+
     def _reset_all(self, _view=None):
         """Settings: forget every chat's cached marks."""
         def run():
@@ -97,6 +130,10 @@ class MessageLoggerPlugin(BasePlugin):
 
         in_background(run)
 
+    def _removed_chats_changed(self):
+        on_ui(self.removed_chats.inject_all)
+        in_background(self.removed_chats.refresh_all)
+
     def purge_cache(self, force=False):
         """Removes cached answers older than the configured lifetime; at most
         once a day (the day turns at local midnight) unless `force`."""
@@ -110,11 +147,14 @@ class MessageLoggerPlugin(BasePlugin):
         for part in (
             "render",
             "chats",
+            "hidden",
             "deletes",
             "delete_dialog",
             "links",
             "menu",
             "comments",
+            "avatars",
+            "removed_chats",
         ):
             if hasattr(self, part):
                 getattr(self, part).uninstall()
@@ -192,6 +232,14 @@ class MessageLoggerPlugin(BasePlugin):
                 subtext="Tell the server about deletions this client sees",
                 icon="msg_report",
             ),
+            Switch(
+                key=SHOW_REMOVED_CHATS,
+                text="Show removed chats",
+                default=DEFAULTS[SHOW_REMOVED_CHATS],
+                subtext="Chats you left or lost stay in the chat list, read-only",
+                icon="msg_archive",
+                on_change=lambda _value: self._removed_chats_changed(),
+            ),
             Text(
                 text="Reset deleted/edited marks",
                 subtext="Forget all cached marks; chats are checked with the server again",

src/media_share.py

@@ -0,0 +1,135 @@
+"""Logger copies read through an account that can, their media shown by any.
+
+The log groups' members (the master, accounts added to them) can read the
+copies; others can't, not even download a copy's media with its file ids.
+So a copy is read through a member signed in here (`reader`: the current
+account when it is one, else the master), and its media downloaded through
+that member into the app's file directories, which all its accounts share:
+the current account then finds the files there.
+"""
+
+import time
+
+import accounts
+from jutil import java_list
+
+_WAIT_SECONDS = 20
+# Larger documents are left to a tap (and fail for a non-member).
+_MAX_DOCUMENT_BYTES = 20 * 1024 * 1024
+
+
+def reader_for(account, log_chat_id, masters=()):
+    """A signed-in member of a log group (marked id): `account` when it is
+    one, else a master, else any; None when none is signed in."""
+    import peers
+
+    channel_id = peers.channel_id_from_marked(log_chat_id)
+    prefer = [account] + [accounts.slot_of(m) for m in masters if m]
+    readers = accounts.with_access(-channel_id, prefer=prefer)
+    return readers[0] if readers else None
+
+
+def _wait(path):
+    deadline = time.monotonic() + _WAIT_SECONDS
+    while not path.exists() and time.monotonic() < deadline:
+        time.sleep(0.25)
+    return path.exists()
+
+
+def _files(photo):
+    """The photo's sizes that are files (inline thumbnails are not)."""
+    return [
+        size
+        for size in java_list(photo.sizes)
+        if getattr(size, "location", None) is not None
+        and int(size.w or 0) > 0
+        and not any(
+            kind in type(size).__name__ for kind in ("Stripped", "PathSize", "Cached")
+        )
+    ]
+
+
+def photo_size(photo, side):
+    """The smallest downloadable size at least `side` px (else the largest):
+    inline thumbnails have no file."""
+    sizes = _files(photo)
+    if not sizes:
+        return None
+    bigger = [size for size in sizes if max(int(size.w), int(size.h)) >= side]
+    if bigger:
+        return min(bigger, key=lambda size: int(size.w))
+    return max(sizes, key=lambda size: int(size.w))
+
+
+def photo(reader, photo, copy, side):
+    """Downloads the photo's size closest to `side` px through `reader` into
+    the shared cache; its path, or None (blocking)."""
+    size = photo_size(photo, side)
+    return _photo_size(reader, photo, size, copy) if size is not None else None
+
+
+def photo_sizes(reader, photo, copy):
+    """Downloads every size of the photo (thumbnails, the viewer's large one)
+    through `reader`; whether the largest is there (blocking)."""
+    sizes = _files(photo)
+    sizes.sort(key=lambda size: int(size.w))
+    paths = [_photo_size(reader, photo, size, copy) for size in sizes]
+    return bool(paths) and paths[-1] is not None
+
+
+def _photo_size(reader, photo, size, copy):
+    from java import jint
+    from org.telegram.messenger import FileLoader, ImageLocation, MessageObject
+
+    loader = FileLoader.getInstance(reader)
+    path = loader.getPathToAttach(size, True)
+    if not path.exists():
+        parent = MessageObject(reader, copy, False, False)
+        loader.loadFile(
+            ImageLocation.getForPhoto(size, photo),
+            parent,
+            "jpg",
+            jint(FileLoader.PRIORITY_HIGH),
+            jint(1),
+        )
+    return str(path.getAbsolutePath()) if _wait(path) else None
+
+
+def document(reader, document, copy):
+    """Downloads a document (up to 20 MB) through `reader` where the app
+    keeps its kind of file; whether it is there (blocking)."""
+    from java import jint
+    from org.telegram.messenger import FileLoader, MessageObject
+
+    if int(getattr(document, "size", 0) or 0) > _MAX_DOCUMENT_BYTES:
+        return False
+    loader = FileLoader.getInstance(reader)
+    path = loader.getPathToAttach(document)
+    if path.exists():
+        return True
+    parent = MessageObject(reader, copy, False, False)
+    loader.loadFile(document, parent, jint(FileLoader.PRIORITY_HIGH), jint(0))
+    return _wait(path)
+
+
+def copy_media(reader, copy):
+    """Downloads a copy's media (photo, document, a rich message's) through
+    `reader` before another account shows it (blocking). Returns (ready,
+    total)."""
+    media = getattr(copy, "media", None)
+    photos, documents = [], []
+    if getattr(media, "photo", None) is not None:
+        photos.append(media.photo)
+    if getattr(media, "document", None) is not None:
+        documents.append(media.document)
+    rich = getattr(copy, "rich_message", None)
+    if rich is not None:
+        photos += java_list(rich.photos)
+        documents += java_list(rich.documents)
+    ready = 0
+    for item in photos:
+        if getattr(item, "sizes", None) is not None:
+            ready += bool(photo_sizes(reader, item, copy))
+    for item in documents:
+        ready += bool(document(reader, item, copy))
+    return ready, len(photos) + len(documents)

src/people.py

@@ -0,0 +1,182 @@
+"""The last known profile of every sender shown from the logger.
+
+The logger stores only a sender's id and name with each message, so the
+app's own user/chat is what carries the avatar, username and the rest. That
+object can be gone (the account never saw the sender again, the app's cache
+was cleared) or useless (a deleted account: no name, no photo). So each time
+a sender is readable here its full TLRPC object is saved, serialized, and it
+is used whenever the app has nothing better.
+
+Resolution, best first: the app's cache, the app's database, the snapshot
+saved here, a stand-in named from the logger, "Deleted".
+"""
+
+import base64
+import threading
+
+import diag
+import store
+
+_lock = threading.Lock()
+_memory = {}  # (account, peer_id) -> TLRPC.User/Chat
+_saved = {}  # (account, peer_id) -> fingerprint of the stored snapshot
+
+
+def _key(account, peer_id):
+    return f"person:{int(account)}:{int(peer_id)}"
+
+
+def _usable_user(user):
+    """A user worth showing: has a name, and isn't an emptied deleted
+    account (which the app keeps with no name and no photo)."""
+    if user is None:
+        return False
+    if type(user).__name__.endswith("userEmpty"):
+        return False
+    name = (getattr(user, "first_name", None) or "") + (getattr(user, "last_name", None) or "")
+    return bool(name.strip())
+
+
+def _usable_chat(chat):
+    if chat is None:
+        return False
+    if type(chat).__name__.endswith(("chatEmpty", "channelForbidden", "chatForbidden")):
+        return bool((getattr(chat, "title", None) or "").strip())
+    return bool((getattr(chat, "title", None) or "").strip())
+
+
+def _fingerprint(obj):
+    photo = getattr(obj, "photo", None)
+    return (
+        getattr(obj, "first_name", None),
+        getattr(obj, "last_name", None),
+        getattr(obj, "title", None),
+        getattr(obj, "username", None),
+        int(getattr(photo, "photo_id", 0) or 0) if photo is not None else 0,
+    )
+
+
+def _serialize(obj):
+    from org.telegram.tgnet import SerializedData
+
+    data = SerializedData(obj.getObjectSize())
+    obj.serializeToStream(data)
+    raw = bytes(data.toByteArray())
+    data.cleanup()
+    return base64.b64encode(raw).decode("ascii")
+
+
+def _deserialize(text, is_user):
+    from org.telegram.tgnet import SerializedData, TLRPC
+
+    raw = base64.b64decode(text)
+    data = SerializedData(raw)
+    try:
+        constructor = data.readInt32(False)
+        cls = TLRPC.User if is_user else TLRPC.Chat
+        return cls.TLdeserialize(data, constructor, False)
+    finally:
+        data.cleanup()
+
+
+def remember(account, peer_id, obj):
+    """Save `obj` (a User for peer_id > 0, a Chat for < 0) when it is worth
+    keeping and differs from the last snapshot."""
+    usable = _usable_user(obj) if peer_id > 0 else _usable_chat(obj)
+    if not usable:
+        return
+    key = (int(account), int(peer_id))
+    mark = _fingerprint(obj)
+    with _lock:
+        _memory[key] = obj
+        if _saved.get(key) == mark:
+            return
+        _saved[key] = mark
+    try:
+        text = _serialize(obj)
+        # After a restart nothing is marked as saved yet: skip the write
+        # when the stored snapshot is already this one.
+        if store.get_meta(_key(account, peer_id)) != text:
+            store.put_meta(_key(account, peer_id), text)
+    except Exception as exc:
+        diag.error("people.remember", exc)
+
+
+def recall(account, peer_id):
+    """The last snapshot saved for the sender, else None."""
+    key = (int(account), int(peer_id))
+    with _lock:
+        if key in _memory:
+            return _memory[key]
+    text = store.get_meta(_key(account, peer_id))
+    if not text:
+        return None
+    try:
+        obj = _deserialize(text, int(peer_id) > 0)
+    except Exception as exc:
+        diag.error("people.recall", exc)
+        return None
+    if obj is None:
+        return None
+    with _lock:
+        _memory[key] = obj
+        _saved[key] = _fingerprint(obj)
+    return obj
+
+
+def _from_app(account, peer_id):
+    """The app's object from its in-memory cache.
+
+    Not its database: getUserSync/getChatSync block on the storage thread,
+    and this runs while a topic's history is being built, so a busy storage
+    queue stalled the whole topic. What the database would add is what the
+    saved snapshot already holds."""
+    from org.telegram.messenger import MessagesController
+
+    controller = MessagesController.getInstance(int(account))
+    if peer_id > 0:
+        return controller.getUser(int(peer_id))
+    return controller.getChat(-int(peer_id))
+
+
+def user(account, peer_id, logged_name=None):
+    """The best User for a sender (peer_id > 0)."""
+    from org.telegram.tgnet import TLRPC
+
+    found = _from_app(account, peer_id)
+    if _usable_user(found):
+        remember(account, peer_id, found)
+        return found
+    saved = recall(account, peer_id)
+    if _usable_user(saved):
+        return saved
+    stand_in = TLRPC.TL_user()
+    stand_in.id = int(peer_id)
+    stand_in.first_name = logged_name or "Deleted"
+    stand_in.flags |= 2
+    return stand_in
+
+
+def chat(account, chat_id, logged_title=None, kind="channel"):
+    """The best Chat for a sender or a removed chat (chat_id > 0)."""
+    from org.telegram.tgnet import TLRPC
+
+    peer_id = -int(chat_id)
+    found = _from_app(account, peer_id)
+    if _usable_chat(found):
+        remember(account, peer_id, found)
+        return found
+    saved = recall(account, peer_id)
+    if _usable_chat(saved):
+        return saved
+    if found is not None:
+        return found
+    if kind == "group":
+        stand_in = TLRPC.TL_chat()
+    else:
+        stand_in = TLRPC.TL_channel()
+        stand_in.broadcast = kind == "channel"
+        stand_in.megagroup = kind != "channel"
+    stand_in.id = int(chat_id)
+    stand_in.title = logged_title or "Chat"
+    return stand_in

src/store.py

@@ -1,5 +1,5 @@
 """What the plugin knows per message: server state (edits, deletion, copy
-pointers) and deletions this client saw itself.
+pointers, hidden marks) and deletions this client saw itself.
 
 Both are cached in SQLite and loaded per chat on first use: a chat opened
 again shows its marks at once, before (or without) the server, and a
@@ -11,6 +11,9 @@ renewed or expired. An expired chat keeps its cache (shown meanwhile) until
 the next successful check replaces it. Independently, cached answers older
 than the configured number of days are removed, checked once a day
 (`purge_old`).
+Messages removed from view (hidden) are marked on the server, so every
+client skips them; a change waits in `hidden_pending` until the server took
+it (and overrides the server's mark meanwhile).
 """
 
 import json
@@ -30,6 +33,7 @@ class State:
         "history",
         "source_date",
         "sender_id",
+        "hidden",
         "raw",
     )
 
@@ -40,6 +44,7 @@ class State:
         self.history = None
         self.source_date = None  # ISO string
         self.sender_id = None
+        self.hidden = False  # removed from view (kept on the server)
         self.raw = None  # the server's answer for it
 
     @classmethod
@@ -51,6 +56,7 @@ class State:
         state.history = data.get("history")
         state.source_date = data.get("source_date")
         state.sender_id = data.get("sender_id")
+        state.hidden = data.get("hidden_at") is not None
         state.raw = data
         return state
 
@@ -92,10 +98,19 @@ def _conn():
         )
         _db.execute("CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT)")
         _db.execute(
-            "CREATE TABLE IF NOT EXISTS hidden ("
+            "CREATE TABLE IF NOT EXISTS hidden_pending ("
-            "account INTEGER, dialog_id INTEGER, message_id INTEGER, at REAL, "
+            "account INTEGER, dialog_id INTEGER, message_id INTEGER, hidden INTEGER, "
-            "PRIMARY KEY (account, dialog_id, message_id))"
+            "at REAL, PRIMARY KEY (account, dialog_id, message_id))"
         )
+        # Before hidden marks were kept on the server: sent there once.
+        if _db.execute(
+            "SELECT name FROM sqlite_master WHERE type='table' AND name='hidden'"
+        ).fetchone():
+            _db.execute(
+                "INSERT OR IGNORE INTO hidden_pending "
+                "SELECT account, dialog_id, message_id, 1, at FROM hidden"
+            )
+            _db.execute("DROP TABLE hidden")
         _db.execute(
             "CREATE TABLE IF NOT EXISTS coverage ("
             "account INTEGER, dialog_id INTEGER, low INTEGER, high INTEGER, at REAL, "
@@ -180,17 +195,17 @@ def put_server_states(account, dialog_id, items):
             diag.error("store.put_server_states", exc)
 
 
-def cached_deleted(account, dialog_id, low, high):
+def cached_deleted(account, dialog_id, low, high, hidden=False):
     """Cached server answers of deleted messages with a logged copy in
-    [low, high] (what a restore needs, without asking the server again)."""
+    [low, high] (what a restore needs, without asking the server again);
+    hidden ones only when `hidden`."""
     with _lock:
         _local_ids(account, dialog_id)
-        hidden = _hidden_ids(account, dialog_id)
         return [
             state.raw
             for message_id, state in _states.get((account, dialog_id), {}).items()
             if low <= message_id <= high
-            and message_id not in hidden
+            and (hidden or not is_hidden(account, dialog_id, message_id))
             and state.raw is not None
             and state.raw.get("deleted_at")
             and state.raw.get("logged")
@@ -323,45 +338,164 @@ def reset(account=None, dialog_id=None):
         return removed
 
 
-_hidden = {}  # (account, dialog_id) -> set of ids the user removed from view
+_pending = None  # (account, dialog_id, message_id) -> hidden, not yet on the server
+_revealed = set()  # (account, dialog_id) showing their hidden messages
 
 
-def _hidden_ids(account, dialog_id):
+def _pending_marks():
-    key = (account, dialog_id)
+    global _pending
-    if key not in _hidden:
+    if _pending is None:
-        found = set()
+        found = {}
         try:
-            rows = _conn().execute(
+            for account, dialog_id, message_id, hidden in _conn().execute(
-                "SELECT message_id FROM hidden WHERE account=? AND dialog_id=?", key
+                "SELECT account, dialog_id, message_id, hidden FROM hidden_pending"
-            ).fetchall()
+            ).fetchall():
-            found = {r[0] for r in rows}
+                found[(account, dialog_id, message_id)] = bool(hidden)
         except Exception as exc:
-            diag.error("store.hidden", exc)
+            diag.error("store.hidden_pending", exc)
-        _hidden[key] = found
+        _pending = found
-    return _hidden[key]
+    return _pending
 
 
-def hide(account, dialog_id, message_ids):
+def set_hidden(account, dialog_id, message_ids, hidden=True):
-    """Messages the user deleted (or removed from view): never shown or
+    """Removes messages from view (or shows them again) here at once; the
-    restored here again."""
+    change waits for the server in `hidden_pending` (see `pending_hidden`)."""
     ids = [int(i) for i in message_ids]
     with _lock:
-        _hidden_ids(account, dialog_id).update(ids)
+        pending = _pending_marks()
+        states = _states.get((account, dialog_id), {})
+        for i in ids:
+            pending[(account, dialog_id, i)] = bool(hidden)
+            state = states.get(i)
+            if state is not None:
+                state.hidden = bool(hidden)
         try:
             db = _conn()
             now = time.time()
             db.executemany(
-                "INSERT OR IGNORE INTO hidden VALUES (?, ?, ?, ?)",
+                "INSERT OR REPLACE INTO hidden_pending VALUES (?, ?, ?, ?, ?)",
-                [(account, dialog_id, i, now) for i in ids],
+                [(account, dialog_id, i, int(bool(hidden)), now) for i in ids],
             )
             db.commit()
         except Exception as exc:
-            diag.error("store.hide", exc)
+            diag.error("store.set_hidden", exc)
+
+
+def pending_hidden():
+    """Hidden marks not on the server yet: {(account, dialog_id, hidden): [ids]}."""
+    with _lock:
+        out = {}
+        for (account, dialog_id, message_id), hidden in _pending_marks().items():
+            out.setdefault((account, dialog_id, hidden), []).append(message_id)
+        return out
+
+
+def hidden_sent(account, dialog_id, message_ids, hidden):
+    """The server took these marks: they are its state now."""
+    ids = [int(i) for i in message_ids]
+    with _lock:
+        pending = _pending_marks()
+        sent = [i for i in ids if pending.get((account, dialog_id, i)) == bool(hidden)]
+        for i in sent:
+            del pending[(account, dialog_id, i)]
+        states = _states.get((account, dialog_id), {})
+        changed = []
+        for i in sent:
+            state = states.get(i)
+            if state is not None and state.raw is not None:
+                state.raw["hidden_at"] = (
+                    state.raw.get("hidden_at") or _now_iso() if hidden else None
+                )
+                state.hidden = bool(hidden)
+                changed.append((account, dialog_id, i, json.dumps(state.raw)))
+        try:
+            db = _conn()
+            db.executemany(
+                "DELETE FROM hidden_pending WHERE account=? AND dialog_id=? AND message_id=?",
+                [(account, dialog_id, i) for i in sent],
+            )
+            db.executemany(
+                "UPDATE states SET data=? WHERE account=? AND dialog_id=? AND message_id=?",
+                [(data, a, d, i) for a, d, i, data in changed],
+            )
+            db.commit()
+        except Exception as exc:
+            diag.error("store.hidden_sent", exc)
+
+
+def _now_iso():
+    import datetime
+
+    return datetime.datetime.now(datetime.timezone.utc).isoformat()
 
 
 def is_hidden(account, dialog_id, message_id):
     with _lock:
-        return int(message_id) in _hidden_ids(account, dialog_id)
+        pending = _pending_marks().get((account, dialog_id, int(message_id)))
+        if pending is not None:
+            return pending
+        state = _states.get((account, dialog_id), {}).get(int(message_id))
+        return state is not None and state.hidden
+
+
+def hidden_deleted(account, dialog_id):
+    """Ids of the chat's deleted messages that are hidden (cached)."""
+    with _lock:
+        _local_ids(account, dialog_id)
+        return [
+            i
+            for i, state in _states.get((account, dialog_id), {}).items()
+            if state.deleted and is_hidden(account, dialog_id, i)
+        ]
+
+
+def reveal(account, dialog_id, on):
+    with _lock:
+        if on:
+            _revealed.add((account, dialog_id))
+        else:
+            _revealed.discard((account, dialog_id))
+
+
+def is_revealed(account, dialog_id):
+    return (account, dialog_id) in _revealed
+
+
+_removed_dialogs = set()  # (account, dialog_id) of removed chats listed
+
+
+def set_removed_dialog(account, dialog_id, listed):
+    """A removed chat listed among the chats (hooks/removed_chats.py): its
+    state comes with its history, the chat is not synced."""
+    with _lock:
+        if listed:
+            _removed_dialogs.add((int(account), int(dialog_id)))
+        else:
+            _removed_dialogs.discard((int(account), int(dialog_id)))
+
+
+def is_removed_dialog(account, dialog_id):
+    return (int(account), int(dialog_id)) in _removed_dialogs
+
+
+def get_meta(key):
+    try:
+        with _lock:
+            row = _conn().execute("SELECT value FROM meta WHERE key=?", (key,)).fetchone()
+        return row[0] if row else None
+    except Exception as exc:
+        diag.error("store.get_meta", exc)
+        return None
+
+
+def put_meta(key, value):
+    try:
+        with _lock:
+            db = _conn()
+            db.execute("INSERT OR REPLACE INTO meta VALUES (?, ?)", (key, value))
+            db.commit()
+    except Exception as exc:
+        diag.error("store.put_meta", exc)
 
 
 def note_own_deletion(account, message_ids):

src/views/comments.py

@@ -33,9 +33,9 @@ read_only = set()
 _lock = threading.Lock()
 
 
-def make_read_only(chat):
+def make_read_only(chat, notice="Viewing deleted comments"):
-    """Swaps a comment screen's input bar for a read-only notice (the app's
+    """Swaps a screen's input bar for a read-only notice (the app's own
-    own bottom overlay, as in chats one can't write to). UI thread."""
+    bottom overlay, as in chats one can't write to). UI thread."""
     from android.view import View
 
     from jutil import field
@@ -47,21 +47,153 @@ def make_read_only(chat):
     text = field(chat, "bottomOverlayText")
     if overlay is None or text is None:
         return
-    for name in ("bottomOverlayChatText", "bottomOverlayStartButton", "bottomOverlayProgress"):
+    # Hide every other control the app may put in the bottom bar. The names
+    # differ per build and per chat kind (a forum/discussion screen shows a
+    # start/join button of its own), so the ones we know are hidden by name
+    # and anything else inside the overlay is hidden by walking it: leaving
+    # one visible is what kept drawing a button over the notice.
+    for name in (
+        "bottomOverlayChat",  # a channel's bar (search, mute/join)
+        "bottomOverlayChatText",
+        "bottomOverlayStartButton",
+        "bottomOverlayProgress",
+        "bottomOverlayImage",
+        "bottomOverlayLinks",
+        "bottomOverlayChatText2",
+        "bottomOverlayChatWaitsReply",
+        "replyButton",
+        "unblockButton",
+        "joinButton",
+        "addToContactsButton",
+        "reportSpamButton",
+        "topViewSeparator",
+    ):
         other = field(chat, name)
-        if other is not None:
+        if other is not None and other is not text:
-            other.setVisibility(View.GONE)
+            try:
+                other.setVisibility(View.GONE)
+            except Exception:
+                pass
+    _hide_siblings(overlay, text, View)
+    _hide_overlay_peers(overlay, View)
     # The app animates the overlay out when the thread has an input bar:
     # stop that, or it ends invisible (alpha 0) after this.
     animation = field(chat, "bottomOverlayAnimation")
     if animation is not None:
         animation.cancel()
-    text.setText("Viewing deleted comments")
+    text.setText(notice)
     text.setVisibility(View.VISIBLE)
+    text.setAlpha(1.0)
     overlay.setAlpha(1.0)
     overlay.setVisibility(View.VISIBLE)
 
 
+def _hide_siblings(overlay, keep, View):
+    """Hide every child of the bottom overlay except the notice label.
+
+    The read-only notice is the app's own `bottomOverlayText`; any other
+    child (a join/start button, a progress spinner) would otherwise be
+    drawn over it. Chaquopy may hand out distinct Python wrappers for the
+    same Java view, so the one to keep is matched with Java equality
+    (`equals`) rather than Python identity."""
+    try:
+        from android.view import ViewGroup
+
+        if not isinstance(overlay, ViewGroup):
+            return
+        for index in range(overlay.getChildCount()):
+            child = overlay.getChildAt(index)
+            if child is None:
+                continue
+            try:
+                if child.equals(keep):
+                    continue
+            except Exception:
+                if child is keep:
+                    continue
+            try:
+                child.setVisibility(View.GONE)
+            except Exception:
+                pass
+        try:
+            keep.bringToFront()
+        except Exception:
+            pass
+    except Exception as exc:
+        diag.error("make_read_only.siblings", exc)
+
+
+def _hide_overlay_peers(overlay, View):
+    """Hide sibling views that share the bottom strip with the overlay.
+
+    Some screens (a forum/discussion ChatActivity) put their action button
+    next to `bottomOverlay` instead of inside it, so hiding the overlay's
+    own children is not enough: the button keeps covering the notice."""
+    try:
+        from android.view import ViewGroup
+
+        parent = overlay.getParent()
+        if parent is None or not isinstance(parent, ViewGroup):
+            return
+        top, bottom = overlay.getTop(), overlay.getBottom()
+        for index in range(parent.getChildCount()):
+            child = parent.getChildAt(index)
+            if child is None:
+                continue
+            try:
+                if child.equals(overlay):
+                    continue
+            except Exception:
+                if child is overlay:
+                    continue
+            try:
+                if child.getVisibility() != View.VISIBLE:
+                    continue
+                # Only the views overlapping the overlay's band: the
+                # message list and the action bar must stay untouched.
+                if child.getBottom() <= top or child.getTop() >= bottom:
+                    continue
+                # The message list spans the whole screen and so overlaps
+                # the band too; hiding it left topics on their loading
+                # placeholders. Only bar-sized views are buttons.
+                band = max(1, bottom - top)
+                if child.getHeight() > band * 2:
+                    continue
+                child.setVisibility(View.GONE)
+            except Exception:
+                pass
+    except Exception as exc:
+        diag.error("make_read_only.peers", exc)
+
+
+def answer(account, request):
+    """The response for a getReplies of a thread opened here, else None."""
+    with _lock:
+        thread = _threads.get(int(request.msg_id))
+    if thread is None or thread[0] != int(account):
+        return None
+    _account, _discussion, messages, users, chats = thread
+    from java.util import ArrayList
+    from org.telegram.tgnet import TLRPC
+
+    offset_id = int(request.offset_id or 0)
+    add_offset = int(request.add_offset or 0)
+    picked = [
+        m for m in messages if not offset_id or add_offset < 0 or int(m.id) < offset_id
+    ]
+    response = TLRPC.TL_messages_messages()
+    response.messages = ArrayList()
+    for message in sorted(picked, key=lambda m: -int(m.id)):
+        response.messages.add(message)
+    response.users = ArrayList()
+    for user in users:
+        response.users.add(user)
+    response.chats = ArrayList()
+    for chat in chats:
+        response.chats.add(chat)
+    return response
+
+
 def _peer(controller, marked):
     """TLRPC peer of a Telethon sender id."""
     from hooks.chat import _android_peer_id
@@ -69,28 +201,17 @@ def _peer(controller, marked):
     return controller.getPeer(_android_peer_id(int(marked)))
 
 
-def _sender_object(controller, marked, name):
+def _sender_object(account, marked, name):
-    """The sender's User/Chat from the account's cache, else a stand-in with
+    """The sender's User/Chat: the app's, else the last snapshot saved while
-    the logged name (avatar drawn from its initials)."""
+    it was readable (name, username, photo), else a stand-in with the
+    logged name (avatar drawn from its initials)."""
+    import people
     from hooks.chat import _android_peer_id
-    from org.telegram.tgnet import TLRPC
 
     peer_id = _android_peer_id(int(marked))
     if peer_id > 0:
-        user = controller.getUser(peer_id)
+        return people.user(account, peer_id, name), None
-        if user is None:
+    return None, people.chat(account, -peer_id, name, "channel")
-            user = TLRPC.TL_user()
-            user.id = peer_id
-            user.first_name = name or "Deleted"
-            user.flags |= 2
-        return user, None
-    chat = controller.getChat(-peer_id)
-    if chat is None:
-        chat = TLRPC.TL_channel()
-        chat.id = -peer_id
-        chat.title = name or "Channel"
-        chat.broadcast = True
-    return None, chat
 
 
 class Comments(History):
@@ -188,7 +309,7 @@ class Comments(History):
             self._into_thread(msg, item, discussion, top, sources)
             if item.get("sender_id"):
                 user, chat = _sender_object(
-                    controller, item["sender_id"], item.get("sender_name")
+                    account, item["sender_id"], item.get("sender_name")
                 )
                 if user is not None:
                     users[user.id] = user

src/views/removed_chat.py

@@ -0,0 +1,571 @@
+"""The history of a removed chat, rebuilt from the logger: the logged messages
+(`/chats/{id}/history`), each from its copy in a log group when an account
+here can read it (header stripped, media included), else from the server's
+text with its media marked unavailable. All of them are deleted for the
+account; they are cached as such (store), so they draw like any deleted
+message.
+
+`History` holds one chat's messages, newest first, and answers the app's
+getHistory/getMessages for it (hooks/removed_chats.py).
+"""
+
+import json
+import threading
+
+import diag
+import media_share
+import peers
+import store
+import tg
+from api import ApiError
+from views.history import _MEDIA_NAMES, _entities, _epoch, _utf16
+
+_PAGES = 3  # history pages (of 100) loaded when the chat opens
+_COPIES_PER_REQUEST = 100
+
+
+def stand_in_user(account, peer_id, name):
+    """The sender's User: the app's, else its last saved snapshot (name,
+    username, photo), else a stand-in with the logged name."""
+    import people
+
+    return people.user(account, peer_id, name)
+
+
+def stand_in_chat(account, chat_id, title, kind):
+    """The app's chat, else its last saved snapshot, else a stand-in."""
+    import people
+
+    return people.chat(account, chat_id, title, kind)
+
+
+class History:
+    def __init__(self, api, account, dialog_id, chat_id):
+        self.api = api
+        self.account = account
+        self.dialog_id = dialog_id
+        self.chat_id = chat_id
+        self.messages = []  # TL messages, newest first
+        self.users = {}
+        self.chats = {}
+        # Forum topics from the server: [{"id", "title", "icon_emoji_id"}],
+        # newest activity first; empty when the chat is no forum.
+        self.topics = []
+        self._topic_of = {}  # message id -> forum topic id (1: General)
+        self._latest_by_topic = {}
+        self.loaded = False
+        self._lock = threading.Lock()
+        self._lock_copies = threading.Lock()
+        self._copy_cache = {}  # (log chat, message id) -> copy (TL message)
+        self._media_done = set()  # (log chat, message id) whose media is local
+        self._pages = {}  # (topic id, before) -> (messages, total)
+        self._downloading = False
+        self._queue = []  # [(items, master)] waiting for their media
+
+    # loading (background)
+    def load(self):
+        """Fetch and build the topic-preview messages once."""
+        with self._lock:
+            if self.loaded:
+                return
+            try:
+                self._load()
+            except Exception as exc:
+                diag.error("removed_chat.load", exc)
+            self.loaded = True
+
+    def _load(self):
+        me = peers.own_user_id(self.account)
+        disk_key = f"chat_head:{self.account}:{self.chat_id}"
+        result = None
+        for attempt in range(2):
+            try:
+                result = self.api.history(me, self.chat_id, 0, limit=1)
+                break
+            except ApiError as exc:
+                diag.log(f"removed chat {self.chat_id} try {attempt + 1}: {exc}")
+        if result:
+            try:
+                store.put_meta(disk_key, json.dumps(result))
+            except Exception as exc:
+                diag.error("removed_chat.head_cache", exc)
+        else:
+            # Unreachable: the last answer saved on this device, so the
+            # chat still opens with its topics and their latest messages.
+            saved = store.get_meta(disk_key)
+            if not saved:
+                return
+            result = json.loads(saved)
+            diag.log(f"removed chat {self.chat_id}: from device cache")
+        result = result or {}
+        self.topics = list(result.get("topics") or [])
+        items = list(result.get("messages") or [])
+        for topic in self.topics:
+            latest = topic.get("latest") or {}
+            if latest and int(latest.get("message_id") or 0) not in {
+                int(i.get("message_id") or 0) for i in items
+            }:
+                items.append(latest)
+        master = result.get("master_tg_id")
+        revealed = store.is_revealed(self.account, self.dialog_id)
+        items = [i for i in items if revealed or not i.get("hidden_at")]
+        store.put_server_states(self.account, self.dialog_id, items)
+        # The topic previews need the logged copies (so media shows instead
+        # of "<kind> unavailable"), but downloading every file here would
+        # block the topic list for as long as the transfers take. Build with
+        # the copies now and fetch their files in the background.
+        self._build(items, master, fetch_copies=True, download=False)
+        self._download_later(items, master)
+
+    def _download_later(self, items, master):
+        """Queue the copies' media for download off the UI path.
+
+        copy_media blocks per file (up to 20s each), so it never runs while a
+        page is being answered. One worker per chat drains the queue: a
+        thread per page made several pages download at once and compete
+        with the UI while the chat was opening."""
+        with self._lock_copies:
+            self._queue.append((list(items), master))
+            if self._downloading:
+                return
+            self._downloading = True
+
+        def run():
+            while True:
+                with self._lock_copies:
+                    if not self._queue:
+                        self._downloading = False
+                        return
+                    batch, batch_master = self._queue.pop(0)
+                try:
+                    self._copies(batch, batch_master, download=True)
+                except Exception as exc:
+                    diag.error("removed_chat.download", exc)
+
+        threading.Thread(target=run, daemon=True).start()
+
+    def _build(self, items, master=None, fetch_copies=True, download=True):
+        copies = self._copies(items, master, download=download) if fetch_copies else {}
+        built = []
+        for item in items:
+            try:
+                built.append(self._message(item, copies.get(self._pointer(item))))
+            except Exception as exc:
+                diag.error("removed_chat.message", exc)
+        self.messages = sorted(built, key=lambda m: -int(m.id))
+        self._latest_by_topic = {}
+        if self.topics:
+            ids = {int(t["id"]) for t in self.topics}
+            by_id = {int(m.id): m for m in self.messages}
+            for item in items:
+                message = by_id.get(int(item["message_id"]))
+                if message is None:
+                    continue
+                topic = int(item.get("source_topic_id") or 0)
+                topic = topic if topic in ids else 1
+                self._topic_of[int(message.id)] = topic
+                self._latest_by_topic.setdefault(topic, message)
+        diag.log(
+            f"removed chat {self.chat_id}: {len(self.messages)} latest messages, "
+            f"{len(self.topics)} topics"
+        )
+
+    def topic_page(self, topic_id, before=0, limit=100):
+        """A topic's logged messages, newest first, with its total count.
+
+        Returns (messages, total): the app pages with limit=1 first, so the
+        total must describe the whole topic, not this slice."""
+        # A page already built is answered from memory: reopening a topic
+        # used to refetch it from the server and rebuild every message.
+        key = (int(topic_id), int(before or 0))
+        with self._lock_copies:
+            page = self._pages.get(key)
+        if page is not None:
+            messages, total = page
+            asked = int(limit or 0)
+            return (messages[:asked] if asked > 0 else messages), total
+        me = peers.own_user_id(self.account)
+        # The app probes a thread with limit=1 before asking for a page.
+        # Passing that straight through fetched a single message and the
+        # topic stayed on its loading placeholders, so a full page is
+        # always fetched and the slice is taken below.
+        want = max(int(limit or 0), 100)
+        disk_key = f"topic_page:{self.account}:{self.chat_id}:{int(topic_id)}:{int(before or 0)}"
+        result = None
+        # Up to two tries: a single slow answer from the server used to leave
+        # the topic empty for good (it looked frozen).
+        for attempt in range(2):
+            try:
+                result = self.api.history(
+                    me, self.chat_id, before, min(100, want), topic_id
+                ) or {}
+                break
+            except ApiError as exc:
+                diag.log(f"removed topic {self.chat_id}/{topic_id} try {attempt + 1}: {exc}")
+        if result is not None and result.get("messages"):
+            try:
+                store.put_meta(disk_key, json.dumps(result))
+            except Exception as exc:
+                diag.error("removed_chat.page_cache", exc)
+        else:
+            # Unreachable or empty: the last answer saved on this device.
+            saved = store.get_meta(disk_key)
+            result = json.loads(saved) if saved else None
+            if result is None:
+                return [], 0
+            diag.log(f"removed topic {self.chat_id}/{topic_id}: from device cache")
+        items = result.get("messages") or []
+        revealed = store.is_revealed(self.account, self.dialog_id)
+        items = [i for i in items if revealed or not i.get("hidden_at")]
+        # Without the logged copies every message falls back to its server
+        # text and its media is marked unavailable: no previews in a topic.
+        # The copies are read now (their media is already in them, by file
+        # id), but the files are downloaded in the background: downloading
+        # them inline held the page for up to 20s per file, so the next
+        # page request waited and the topic sat on its placeholders.
+        master = result.get("master_tg_id")
+        copies = self._copies(items, master, download=False)
+        self._download_later(items, master)
+        built = []
+        for item in items:
+            try:
+                built.append(self._message(item, copies.get(self._pointer(item))))
+            except Exception as exc:
+                diag.error("removed_chat.topic_message", exc)
+        ordered = sorted(built, key=lambda m: -int(m.id))
+        total = len(ordered)
+        if ordered:
+            with self._lock_copies:
+                self._pages[key] = (ordered, total)
+        # Answer with at most what was asked for: the app drops a reply that
+        # holds more messages than its limit, which left the topic on its
+        # placeholders even though the history had arrived.
+        asked = int(limit or 0)
+        if asked > 0:
+            ordered = ordered[:asked]
+        return ordered, total
+
+    @staticmethod
+    def _pointer(item):
+        logged = item.get("logged") or {}
+        if not logged.get("chat_id") or not logged.get("message_id"):
+            return None
+        return int(logged["chat_id"]), int(logged["message_id"])
+
+    def _copies(self, items, master, download=True):
+        """Logged copies by (log chat, message), read through an account here
+        that can, per log group."""
+        by_group = {}
+        for item in items:
+            pointer = self._pointer(item)
+            if pointer is not None:
+                by_group.setdefault(pointer[0], set()).add(pointer[1])
+        found = {}
+        masters = [master, *self.api.masters()]
+        for log_chat, ids in by_group.items():
+            # Copies already read are reused: a reopened chat or topic would
+            # otherwise ask Telegram for every logged copy again.
+            with self._lock_copies:
+                missing = [i for i in ids if (log_chat, i) not in self._copy_cache]
+                for i in ids:
+                    cached = self._copy_cache.get((log_chat, i))
+                    if cached is not None:
+                        found[(log_chat, i)] = cached
+            if not missing:
+                if download:
+                    self._download_copies(log_chat, ids, master, found)
+                continue
+            channel_id = peers.channel_id_from_marked(log_chat)
+            reader = media_share.reader_for(self.account, log_chat, masters)
+            if reader is None:
+                # No signed-in account is a member of this log group, so its
+                # copies (and their media) cannot be read at all: every
+                # message falls back to "<kind> unavailable".
+                diag.log(f"copies: no reader for log group {log_chat} ({len(missing)} messages)")
+                continue
+            ordered = sorted(missing)
+            fetched = 0
+            for start in range(0, len(ordered), _COPIES_PER_REQUEST):
+                chunk = ordered[start : start + _COPIES_PER_REQUEST]
+                for message_id, copy in _fetch(reader, -channel_id, chunk).items():
+                    fetched += 1
+                    found[(log_chat, message_id)] = copy
+                    with self._lock_copies:
+                        self._copy_cache[(log_chat, message_id)] = copy
+            diag.log(
+                f"copies: log group {log_chat} reader={reader} "
+                f"asked={len(ordered)} got={fetched}"
+            )
+            if download:
+                self._download_copies(log_chat, ids, master, found)
+        return found
+
+    def _download_copies(self, log_chat, ids, master, found):
+        """Download each copy's media once (blocking; background only).
+
+        The media is fetched through a log-group member by the copy's own
+        file reference. Files already downloaded are skipped, so reopening a
+        chat costs nothing; a file that failed is retried on a later open."""
+        reader = media_share.reader_for(self.account, log_chat, [master, *self.api.masters()])
+        if reader is None:
+            return
+        ready = failed = 0
+        for i in ids:
+            key = (log_chat, i)
+            copy = found.get(key)
+            if copy is None or key in self._media_done:
+                continue
+            got, total = media_share.copy_media(reader, copy)
+            if got >= total:
+                self._media_done.add(key)
+                ready += got
+            else:
+                failed += total - got
+        if ready or failed:
+            diag.log(f"media: log group {log_chat} ready={ready} failed={failed}")
+
+    # building
+
+    def _message(self, item, copy):
+        from org.telegram.messenger import MessagesController
+        from org.telegram.tgnet import TLRPC
+
+        from hooks.chat import _android_peer_id, build_from_copy
+
+        controller = MessagesController.getInstance(self.account)
+        message_id = int(item["message_id"])
+        if copy is not None and not isinstance(copy, TLRPC.TL_messageEmpty):
+            msg = build_from_copy(
+                self.account, self.dialog_id, message_id, item, copy, True
+            ).messageOwner
+        else:
+            msg = from_text(self.account, self.dialog_id, item)
+        sender = item.get("sender_id")
+        if sender:
+            peer_id = _android_peer_id(int(sender))
+            if peer_id > 0:
+                user = stand_in_user(self.account, peer_id, item.get("sender_name"))
+                self.users[int(user.id)] = user
+            else:
+                chat = stand_in_chat(self.account, -peer_id, item.get("sender_name"), "channel")
+                self.chats[int(chat.id)] = chat
+        topic = int(item.get("source_topic_id") or 0)
+        if topic and self.topics:
+            # In a forum topic: the reply header names it (its start message
+            # id), with the replied message when there is one.
+            reply = TLRPC.TL_messageReplyHeader()
+            reply.forum_topic = True
+            reply.flags |= 8
+            target = int(item.get("reply_to") or 0)
+            reply.reply_to_msg_id = target or topic
+            reply.flags |= 16
+            if target and target != topic:
+                reply.reply_to_top_id = topic
+                reply.flags |= 2
+            msg.reply_to = reply
+            msg.flags |= 8
+        elif item.get("reply_to"):
+            reply = TLRPC.TL_messageReplyHeader()
+            reply.reply_to_msg_id = int(item["reply_to"])
+            reply.flags |= 16
+            msg.reply_to = reply
+            msg.flags |= 8
+        return msg
+
+    # forum topics
+
+    def topic_of(self, message):
+        return self._topic_of.get(int(message.id), 1)
+
+    def topic_messages(self, topic_id):
+        return [self._latest_by_topic[topic_id]] if int(topic_id) in self._latest_by_topic else []
+
+    def forum_topics(self, offset_topic=0, limit=100, only=None):
+        """messages.getForumTopics' answer: General first, then the topics by
+        newest message; each with its newest message. Every topic asked for
+        (`only`) is answered, empty ones too, or the app asks again forever."""
+        from java.util import ArrayList
+        from org.telegram.tgnet import TLRPC
+        entries = list(self.topics)
+        if not any(int(e["id"]) == 1 for e in entries):
+            entries.insert(0, {"id": 1, "title": "General"})
+
+        def _recency(entry):
+            """Newest logged activity in the topic (0 when nothing logged)."""
+            latest = entry.get("latest") or {}
+            return _epoch(latest.get("source_date")) if latest else 0
+
+        # Telegram orders a forum by the most recent topic, General first.
+        # The logger records no pinned flag or order (the server sends only
+        # id/title/icon/latest), so recency is the ordering that can be
+        # rebuilt faithfully; without this the topics came out in the
+        # server's arbitrary order.
+        if only is None:
+            general = [e for e in entries if int(e["id"]) == 1]
+            rest = [e for e in entries if int(e["id"]) != 1]
+            rest.sort(key=_recency, reverse=True)
+            entries = general + rest
+        if only is not None:
+            known = {int(e["id"]) for e in entries}
+            entries = [e for e in entries if int(e["id"]) in only] + [
+                {"id": i, "title": f"Topic {i}"} for i in only if i not in known
+            ]
+        elif offset_topic:
+            ids = [int(e["id"]) for e in entries]
+            if int(offset_topic) in ids:
+                entries = entries[ids.index(int(offset_topic)) + 1 :]
+        entries = entries[: max(1, int(limit or 100))]
+        response = TLRPC.TL_messages_forumTopics()
+        response.topics = ArrayList()
+        response.messages = ArrayList()
+        tops = set()
+        for entry in entries:
+            topic_id = int(entry["id"])
+            messages = self.topic_messages(topic_id)
+            if not messages and only is None:
+                continue
+            topic = TLRPC.TL_forumTopic()
+            topic.id = topic_id
+            topic.title = str(entry.get("title") or f"Topic {topic_id}")
+            icon = entry.get("icon_emoji_id")
+            if icon:
+                topic.icon_emoji_id = int(icon)
+                topic.flags |= 1
+            topic.icon_color = 0x6FB9F0
+            if not messages:
+                # Nothing logged in it: an empty stand-in, so the app has a
+                # top message and does not ask for the topic again.
+                empty = TLRPC.TL_message()
+                empty.id = topic_id
+                empty.peer_id = self.messages[0].peer_id if self.messages else None
+                empty.message = ""
+                messages = [empty]
+            # The newest message dates the topic: messages are newest-first,
+            # so messages[-1] is the OLDEST and dating the topic by it put
+            # active topics at the bottom of the forum.
+            topic.date = int(messages[0].date)
+            topic.top_message = int(messages[0].id)
+            # Deleted topics are read-only snapshots; don't let stale topic
+            # unread state create a badge in the topic list.
+            topic.read_inbox_max_id = topic.top_message
+            topic.read_outbox_max_id = topic.top_message
+            if hasattr(topic, "unread_count"):
+                topic.unread_count = 0
+            if hasattr(topic, "unread_mentions_count"):
+                topic.unread_mentions_count = 0
+            # The topic's creator shown in the list is the newest message's
+            # sender, matching the preview drawn from messages[0].
+            if messages[0].from_id is not None:
+                topic.from_id = messages[0].from_id
+            response.topics.add(topic)
+            if topic.top_message not in tops:
+                tops.add(topic.top_message)
+                response.messages.add(messages[0])
+        response.count = response.topics.size()
+        response.pts = 0
+        response.users = ArrayList()
+        for user in self.users.values():
+            response.users.add(user)
+        response.chats = ArrayList()
+        for chat in self.chats.values():
+            response.chats.add(chat)
+        return response
+
+    # answering the app's requests
+
+    def response(self, picked, total=None, offset=None):
+        """A getHistory/getReplies answer.
+
+        TL_messages_messages is what the app expects here: switching to
+        TL_messages_channelMessages (to carry a count) made every topic
+        hang on its loading placeholders, so the plain form is kept."""
+        from java.util import ArrayList
+        from org.telegram.tgnet import TLRPC
+
+        response = TLRPC.TL_messages_messages()
+        response.messages = ArrayList()
+        for message in picked:
+            response.messages.add(message)
+        response.users = ArrayList()
+        for user in self.users.values():
+            response.users.add(user)
+        response.chats = ArrayList()
+        for chat in self.chats.values():
+            response.chats.add(chat)
+        return response
+
+    def page(self, offset_id, add_offset, limit, messages=None):
+        """getHistory's slice: newest first, from `offset_id` (exclusive; 0:
+        the newest) moved by `add_offset` (negative: newer ones too)."""
+        messages = self.messages if messages is None else messages
+        if offset_id:
+            start = next(
+                (i for i, m in enumerate(messages) if int(m.id) < offset_id),
+                len(messages),
+            )
+        else:
+            start = 0
+        start = max(0, start + add_offset)
+        return messages[start : start + max(1, limit)]
+
+    def by_ids(self, ids):
+        wanted = set(int(i) for i in ids)
+        return [m for m in self.messages if int(m.id) in wanted]
+
+
+def from_text(account, dialog_id, item):
+    """A logged message from the server's text; each media is marked
+    unavailable after it."""
+    from org.telegram.messenger import MessagesController
+    from org.telegram.tgnet import TLRPC
+
+    from hooks.chat import _android_peer_id
+
+    controller = MessagesController.getInstance(account)
+    msg = TLRPC.TL_message()
+    msg.id = int(item["message_id"])
+    msg.dialog_id = dialog_id
+    msg.peer_id = controller.getPeer(dialog_id)
+    sender = item.get("sender_id")
+    msg.from_id = (
+        controller.getPeer(_android_peer_id(int(sender))) if sender else msg.peer_id
+    )
+    msg.flags |= 256
+    msg.date = _epoch(item.get("source_date"))
+    if int(item.get("edit_count") or 0) > 0:
+        msg.edit_date = _epoch(item.get("edit_date")) or msg.date
+        msg.flags |= 32768
+    text = item.get("text") or ""
+    entities = _entities(item.get("entities") or [])
+    notes = [
+        f"🚫 {_MEDIA_NAMES.get(kind, 'Media')} unavailable"
+        for kind in item.get("media") or []
+    ]
+    if notes:
+        note = "\n".join(notes)
+        start = _utf16(text) + (2 if text else 0)
+        text = (text + "\n\n" if text else "") + note
+        italic = TLRPC.TL_messageEntityItalic()
+        italic.offset = start
+        italic.length = _utf16(note)
+        entities.add(italic)
+    msg.message = text or "(empty)"
+    msg.entities = entities
+    msg.out = sender is not None and int(sender) == peers.own_user_id(account)
+    return msg
+
+
+def _fetch(reader, dialog_id, ids):
+    """Messages `ids` of a log group through `reader` (blocking)."""
+    done = threading.Event()
+    box = {}
+
+    def callback(found, error):
+        if found is None:
+            diag.log(f"removed chat copies unreadable: {error}")
+        box["found"] = found or {}
+        done.set()
+
+    tg.get_messages(reader, dialog_id, ids, callback)
+    done.wait(20)
+    return box.get("found") or {}

tools/verify_bundle.py

@@ -314,7 +314,7 @@ def main(argv=None) -> int:
         store.put_server_states(0, -7, [{"message_id": 1, "edit_count": 1}])
         store.put_server_states(0, -8, [{"message_id": 1, "edit_count": 1}])
         store.mark_deleted(0, -7, [2])
-        store.hide(0, -7, [3])
+        store.set_hidden(0, -7, [3])
         check(
             "reset forgets one chat's marks, keeps others and hidden ones",
             store.reset(0, -7) == 2
@@ -322,6 +322,24 @@ def main(argv=None) -> int:
             and store.get(0, -8, 1) is not None
             and store.is_hidden(0, -7, 3),
         )
+        store.put_server_states(
+            0, -9, [{"message_id": 4, "deleted_at": "x", "logged": {"chat_id": 1}}]
+        )
+        shown = len(store.cached_deleted(0, -9, 1, 10)) == 1
+        store.set_hidden(0, -9, [4])
+        waiting = store.pending_hidden().get((0, -9, True))
+        skipped = store.cached_deleted(0, -9, 1, 10) == []
+        store.hidden_sent(0, -9, [4], True)
+        check(
+            "hidden marks wait for the server, then are its state",
+            shown
+            and waiting == [4]
+            and skipped
+            and (0, -9, True) not in store.pending_hidden()
+            and store.is_hidden(0, -9, 4)
+            and len(store.cached_deleted(0, -9, 1, 10, True)) == 1,
+            f"waiting={waiting} skipped={skipped}",
+        )
         for name in ("cached_deleted", "coverage", "set_coverage", "expire_coverage", "replace_window"):
             check(f"store.{name} exists", callable(getattr(store, name, None)))