Undocumented change to how we handle Response data

[pavlis]

Somewhere in the recent flurry of changes made with the AI to fix some old problem an unexpected change happened. Previously the method Database.save_inventory saved the response data for a channel to an attribute with the key "serialized_channel_data". It still does that but the (default) format it uses changed completely. Previously the obspy Response object was serialized with pickle.dumps. It seems we now have a new module in mspasspy.db.serialization called encode_inventory. Apparently that converts the response data back to StationXML using an obspy function and saves that as the value associated with "serialized_channel_data". The new method to restore "serialized_channel_data" from a document in channel is this code snippet:

from mspasspy.db.serialization import decode_response
# example just pulling first document in channel
doc = db.channel.find_one()
response_doc = doc["serialized_channel_data"]
resp = decode_response(response_doc)

Poking around the code it seems there is some way to make the backward compatible to handle data stored previously with pickle. I haven't really dug into that as I don't think it is that important. I would always recommend most uses should use web services to build the channel metadata anyway to assure it is always as current as possible since thing do get changed there. The only way that would enter then is if reading old channel collection data saved in something like a json dump.

I am posting this as there are two things we need to do to follow up on this:

1. I haven't checked but presume the user manual is out of date on this topic. Somewhere, I know, it discusses handling response data and the whole issue of using web services top populate the channel and site collections. We need to clean any misinformation there.
2. The mspass_tutorials are a bigger problem as unless something was done there is misinformation there too.

The good news is it looks like the calib.py module addressed this change and that is not part of this issue. For the record, that comes up because the "calib" attribute to scale data from counts to nm/s is contained in the Response object that is created by decode_response. It appears that was changed when save_inventory was changed,

This not a high priority issue as it seems to be restricted to a documentation problem but we do need to address it at some point. For now just keeping it as an issue will address the misinformation.

[pavlis]

I see something else this code change did that is worth discussion. I see the Inventory object is serialized into the site collection. I think that is a really bad idea for three reasons:

1. That monstrosity seriously bloats that collection. It increases it size by at least one order of magnitude.
2. It is a collision in concept. The obspy.Inventory object was not designed to serve that role. It was designed to be a combination of what we do with site and channel.
3. The original design of the site collection was to serve as the Metadata for Seismogram objects. The baggage of seed station codes and channel responses do not belong in site. The model should be that Seismogram objects have not concept of Response as by the time you get them into that form they should be normalized correctly. Furthermore, inventory orientation data is meaningless to a Seismogram object.

I think we should remove the code from save_inventory that puts data to the key "serialized_inventory".

[wangyinz]

I traced the relevant history. There are two separate issues here:

1. The change to channel.serialized_channel_data is recent and comes from PR Add versioned BSON persistence with legacy pickle compatibility #989. That PR intentionally changed new Response writes from pickle.dumps(channel) to the versioned encode_response(...) representation while retaining legacy pickle read compatibility. So the Response-format change itself is not an accidental regression, but the user-facing documentation/tutorials were not updated to match it.

2. The site.serialized_inventory behavior is tied to a much older inconsistency. Historically read_inventory expected serialized_inventory, but save_inventory stopped writing it years ago. Issue Replace database pickle payloads with versioned BSON-native documents #841 identified that broken public round trip, and PR Add versioned BSON persistence with legacy pickle compatibility #989 restored the field in order to make save_inventory/read_inventory round-trip again. However, that also reintroduced the full Inventory payload into the site collection.

Given the latest comment here, I agree the current site.serialized_inventory design should be removed. The site collection should remain receiver metadata for Seismogram normalization and should not carry an embedded Inventory/Response object graph. I am preparing a PR that removes serialized_inventory writes from save_inventory and adjusts the affected tests/API behavior accordingly, while leaving the new Response codec in channel.serialized_channel_data intact.

[wangyinz]

PR #1038 implements the requested change: save_inventory no longer writes serialized_inventory into site documents, while the versioned Response payload in channel.serialized_channel_data is left unchanged. The tests now explicitly assert that newly saved site records do not contain serialized_inventory. Existing read_inventory decoding is retained so databases that already contain serialized_inventory remain readable.