Successfully migrated from ZHA to Z2M without re-pairing

[seidtgeist]

@Koenkk First of all, love the project! Thank you! I'm so glad to be able to set light gradients on my Signe again, and the software updates are a welcome plus (for no reason other than feeling up to date).

So, I've had a fun night and automated the device migration part with a script:

1. Stop Z2M
2. Get list of devices and manufacturer name from ZHA zigbee.db
3. Find device by IEEE to get friendly name from HA config
4. For every device, create the minimum viable JSON database.db entry, e.g.:

   {
     "id": 3,                            // next highest id
     "type": "EndDevice",                // this will be updated by interview process
     "ieeeAddr": "0x0017880110409015",   // reformatted ieee from zigbee.db
     "nwkAddr": 19243,                   // nwk from zigbee.db
     "manufId": 4107                     // endpoint attribute mapped from zigbee.db, mapped via Z2M's cluster.ts
   }

5. Append one entry per line for every device in zigbee2mqtt/database.db
6. For friendly device names, create this YAML snippet for zigbee2mqtt/configuration.yaml:

   devices:
     '0x0017880110409015':
       friendly_name: Office Floor
     # ...

7. Start Z2M
8. Start interview process for each device
9. Done. No need to enable permit join, re-pair or factory-reset anything in my case. Z2M will fill in the device endpoints, cluster data, etc.

This could be automated further for a full ZHA import:

1. Extract PAN, Extended PAN, Network Key and Channel from ZHA network config
2. Follow the above steps for the device database
3. Trigger interview process for all devices via MQTT

I guess you know all this already, but I've read quite a few forum discussion and watched youtube videos where people crawl through their homes, re-pairing all of their devices manually 🤭

I could write this as a standalone TS script if you're interested?

[Koenkk]

Interesting! Had a chat with @Nerivec about this, it should be easy to add an automatic interview when an unknown device sends a message.

So the procedure would be:

• Make sure network settings are the same (key, channel, panid and ext panid). Can these be retrieved easily?
• Start z2m, trigger every device such that it sends a message -> device gets interviewed

[seidtgeist]

• Make sure network settings are the same (key, channel, panid and ext panid). Can these be retrieved easily?

Yes. Want me to write a function which reads the relevant ZHA settings and returns them?

• Start z2m, trigger every device such that it sends a message -> device gets interviewed

That's right.

No more pair-crawling :D

[Koenkk]

Yes. Want me to write a function which reads the relevant ZHA settings and returns them?

Yes, would be cooler if it could also generate a part of the z2m configuration.yaml!

[Nerivec]

key, channel, panid and ext panid

That's the stuff from the coordinator_backup.json, so, should already be fine on that point. Just need to copy it over from ZHA (I think you can even download it from ZHA?).

The newer coordinator_backup.json format (once finalized) should have an independent list of ieee/nwk pairs, which would allow to fill in the database with the minimal info, from Controller, if the result of adapter start was restore and the database is empty. Then we'd have to trigger a round of (throttled) interviews for routers (easy enough). Then trigger interviews for end devices on reception of a message (should prevent too many failures because of sleep, keep manual interactions to a minimum). Since the database would be filled with minimal info, we can simply use the interviewed/interviewing fields to dictate behavior on this.

Just a quick thought, but something along these lines should be doable.
Definitely on the TODO list 😉

[JayGoldberg]

In my case the downloaded ZHA backup 2024-11-08T05-25-59.735Z.json (Download Backup in the ZHA Configure window) contains keys and IDs but not friendly names, etc. It seems to be a radio backup?

As I understand, all of this is already on the radio so you should be able to just stop ZHA, start Z2M and point it at the same radio? As @thargy alludes to in this discussion.

[thargy]

Only the IEEE and Network address is held on each radio to confirm it is registered with the network. The device's manufacturer ID, model, clusters, etc. can be interviewed. What isn't available is a friendly name/entity ID, so either that data needs to be grabbed from ZHA's config, or supplied manually.

There's an argument for Z2m to populate the IEEEAddress as an attribute against the entity in HA as it would be useful for restoring an association. I've also asked about populating a list of child entities for groups, or at least supply a GroupID.

[seidtgeist]

What to do after coming home from a nice party? Feed the cats and deliver the code, of course!

Note: The most straightforward way would be to instruct users to download a backup from ZHA and then disable it. However, you could probably also just curl the backup from the other extension.

example.ts

import { convertZHAtoZ2M } from './slurpbee.ts'

// Excerpt of actual ZHA backup.json
const zhaBackup = {
    "version": 1,
    "network_info": {
        "extended_pan_id": "04:08:15:16:23:42:13:37",
        "pan_id": "78B3",
        "channel": 25,
        "network_key": {
            "key": "04:08:15:16:23:42:13:37:04:08:15:16:23:42:13:37"
        }
    }
};

// There might be a better way of getting this data and I don't know it
const haRegistry = JSON.parse(await Deno.readTextFile(
    'path/to/.storage/core.device_registry'
));

// Convert to Z2M format and then to YAML
const z2mConfig = convertZHAtoZ2M(zhaBackup, haRegistry);
const yamlOutput = jsyaml.dump(z2mConfig, {
    indent: 2,
    lineWidth: -1,
    noRefs: true,
});

console.log(yamlOutput);

advanced:
  pan_id: 30899
  ext_pan_id:
    - 4
    - 8
    - 21
    - 22
    - 35
    - 66
    - 19
    - 55
  channel: 25
  network_key:
    - 4
    - 8
    - 21
    - 22
    - 35
    - 66
    - 19
    - 55
    - 4
    - 8
    - 21
    - 22
    - 35
    - 66
    - 19
    - 55
devices:
  0xe0:79:8d:ff:fe:cf:c3:09:
    friendly_name: 'Generic Zigbee Coordinator (EZSP)' # note: not sure the coordinator should be imported, probably not :) i didn't do it.
  0xb4:35:22:ff:fe:56:c2:ef:
    friendly_name: Boiler Room Mirror Right
  0xb4:35:22:ff:fe:c6:b1:96:
    friendly_name: Boiler Room Mirror Left
  0x00:17:88:01:0b:7f:38:26:
    friendly_name: Boiler Room Ceiling

slurpbee.ts

import yaml from 'js-yaml';

interface ZHABackup {
    version: number;
    network_info: {
        extended_pan_id: string;
        pan_id: string;
        channel: number;
        network_key: {
            key: string;
        };
    };
}

interface HADevice {
    identifiers: [string, string][];
    name_by_user: string | null;
    name: string;
    area_id?: string | null;
}

interface HARegistry {
    data: {
        devices: HADevice[];
    };
}

interface Z2MConfig {
    advanced: {
        pan_id: number;
        ext_pan_id: number[];
        channel: number;
        network_key: number[];
    };
    devices: {
        [key: string]: {
            friendly_name: string;
        };
    };
}

function hexStringToNumbers(hexString: string): number[] {
    return hexString.split(':').map(hex => parseInt(hex, 16));
}

/**
 * Extracts ZHA devices from Home Assistant's device registry
 */
function extractZHADevices(registry: HARegistry): Record<string, string> {
    const devices: Record<string, string> = {};

    registry.data.devices
        .filter(device =>
            device.identifiers?.length > 0 &&
            device.identifiers[0][0] === 'zha'
        )
        .forEach(device => {
            const ieee = device.identifiers[0][1];
            // Prefer user-defined name, fall back to default name
            const name = device.name_by_user || device.name;
            devices[ieee] = name;
        });

    return devices;
}

function convertZHAtoZ2M(zhaBackup: ZHABackup, registry: HARegistry): Z2MConfig {
    const zhaDevices = extractZHADevices(registry);

    return {
        advanced: {
            pan_id: parseInt(zhaBackup.network_info.pan_id, 16),
            ext_pan_id: hexStringToNumbers(zhaBackup.network_info.extended_pan_id),
            channel: zhaBackup.network_info.channel,
            network_key: hexStringToNumbers(zhaBackup.network_info.network_key.key),
        },
        devices: Object.fromEntries(
            Object.entries(zhaDevices).map(([ieee, name]) => [
                `0x${ieee}`,  // Ensure IEEE has 0x prefix
                { friendly_name: name }
            ])
        )
    };
}

export { convertZHAtoZ2M, type ZHABackup, type Z2MConfig };

[thargy]

Note you only need the IEEE and Network addresses. You can set the "type" to "Unknown". The interview will grab the correct type and everything else (including the Manufacturer ID).

For coordinator's that support backups, the IEEE and Network addresses should be retrieved from the coordinator as that is the definitive source of truth. For a ZHA migration, you only need to grab the IEEE and friendly name mapping.

I believe all the relevant state should also be queried from the device.

Most importantly, you have to batch the interviews otherwise you are likely to crash the network/coordinator. In my testing a maximum of 3/4 concurrent interviews is safe.

[JayGoldberg]

Does Z2H mean ZHA or Z2M? I think this was a typo of disabling ZHA, since the next step is to copy the ZHA zigbee.db.

[seidtgeist]

Z2M, thanks for catching this! We want it to be stopped before editing the database on disk.

[JayGoldberg]

I was able to download zigbee.db on a HAOS OVA install by using the File editor addon.

Then dumping some attributes from it there are quite a few tables and they are versioned:

$ echo ".schema" | sqlite3 zigbee.db |grep "CREATE TABLE"
...
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE devices_v13 (
CREATE TABLE endpoints_v13 (
CREATE TABLE clusters_v13 (
CREATE TABLE attributes_cache_v13 (
CREATE TABLE neighbors_v13 (
CREATE TABLE routes_v13 (
CREATE TABLE node_descriptors_v13 (
CREATE TABLE groups_v13 (
CREATE TABLE group_members_v13 (
CREATE TABLE relays_v13 (
CREATE TABLE unsupported_attributes_v13 (
CREATE TABLE network_backups_v13 (

I think we're only interested in devices

$ sqlite3 -header -column zigbee.db "SELECT * FROM devices_v13;"
ieee                     nwk    status  last_seen       
-----------------------  -----  ------  ----------------
00:12:4b:00:25:8c:2e:01  0      2       1731087968.03853
a4:c1:38:ff:f0:68:ee:84  36015  2       1727419863.37843
a4:c1:38:99:1f:8f:b1:74  38073  2       1731085873.37684
28:2c:02:bf:ff:ed:7f:50  19380  2       1731088878.55636
28:2c:02:bf:ff:ed:ff:39  36867  2       1731088891.67052
28:2c:02:bf:ff:ed:3a:ff  54497  2       1731088883.11546
a4:c1:38:03:8b:85:e2:fa  9725   2       1726708778.38752
28:2c:02:bf:ff:ed:a7:ab  38581  2       1731088855.53571
28:2c:02:bf:ff:e3:11:45  47669  2       1730996825.1379 
a4:c1:38:4e:36:1d:2c:90  17650  2       1731080608.45794
a4:c1:38:92:34:e5:72:f5  58464  2       1731084464.99849
28:2c:02:bf:ff:e8:fd:45  10249  2       1731087959.9287 
a4:c1:38:12:0d:3a:64:93  46338  2       1731088896.81854
a4:c1:38:a6:ff:b9:d3:b8  55145  2       1731084162.01582
a4:c1:38:1b:37:9f:a2:3d  53758  2       1731085470.95818

There's not a good way of exporting devices from HA and I don't have access to .storage. I'm looking at https://community.home-assistant.io/t/export-devices-and-entities/259465 to get the friendly names to build a basic JSON entry with ieee & nwk.

[koying]

Thanks for this.

Note that the ext_pan_id is in reverse order in the Z2M configuration vs. a ZHA backup for me (no clue who is right)

ZHA backup

    "network_info": {
        "extended_pan_id": "7e:51:d3:4f:64:50:5a:6e",

Z2M config

  ext_pan_id:
    - 110
    - 90
    - 80
    - 100
    - 79
    - 211
    - 81
    - 126

[Nerivec]

Big-endian, little-endian, it's not a matter of "right", but "right where it's used". Z2M adapts it as necessary throughout the code, so does ZHA. 😉

[koying]

They might, but my coordinator (yellow) was not happy to see a different ext_pan_id. Something in the like of "was configured with", with the ZHA ext_pan_id in reverse order if I specify the ext_pan_id in config the same way as ZHA.

[Nerivec]

ZHA stores them one way, and Z2M the other, so you indeed need to reverse them when switching from one to the other, in the file that is. The various systems in Z2M usually take care of this conversion when importing a coordinator_backup.json into Z2M, but the process here is a bit more "hands-on", so some conversions have to be made manually.
Hopefully we can build in support for most of this soon 😉

[ofcaah]

ZHA stores them one way, and Z2M the other, so you indeed need to reverse them when switching from one to the other, in the file that is. The various systems in Z2M usually take care of this conversion when importing a coordinator_backup.json into Z2M, but the process here is a bit more "hands-on", so some conversions have to be made manually. Hopefully we can build in support for most of this soon 😉

Hey. First of all, thank you for all your contributions. I'm also in the "ZHA continues its progress down the drain, and I don't want my network to sink with the ship" boat. I was thinking about the "hands-on way" migration this weekend, but after reading your post, maybe I should postpone it for some time, if not only to make things easier for myself, then to provide a "it worked/it borked" datapoint on the less "hands-on" approach. Question is, in your estimation, how much longer would I have to suffer ZHA restarting itself/doing other unwanted things before even a beta-quality procedure is implemented? ;)

PS. Looking at things, I think implementing such procedure will result in influx of new Z2M users :>

[Nerivec]

ZHA currently does not follow the open coordinator backup format, which makes it hard to derive a proper logic from it (would have to walk every single version from the moment they deviated from the format to adapt to every change, a monstruous task).
It should be easier once the new format is put in place, Z2M can add support for it, and then migrating from Z2M<>ZHA should be simpler. The new format will also provide all the necessary information, whereas now, it would require parsing databases (same problem as above: past/future changes would have to be supported...).
No ETA can be given at this time.

[bobobo1618]

Tried following this but something is unhappy:

error: 	z2m: Error: network commissioning timed out - most likely network with the same panId or extendedPanId already exists nearby (Error: AREQ - ZDO - stateChangeInd after 60000ms
    at Object.start (/app/node_modules/zigbee-herdsman/src/utils/waitress.ts:59:23)
    at ZnpAdapterManager.beginCommissioning (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:370:31)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:91:21)
    at ZStackAdapter.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:158:16)
    at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:137:29)
    at Zigbee.start (/app/lib/zigbee.ts:69:27)
    at Controller.start (/app/lib/controller.ts:161:27)
    at start (/app/index.js:154:5))
    at ZnpAdapterManager.beginCommissioning (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:372:23)
    at ZnpAdapterManager.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/manager.ts:91:21)
    at ZStackAdapter.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:158:16)
    at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:137:29)
    at Zigbee.start (/app/lib/zigbee.ts:69:27)
    at Controller.start (/app/lib/controller.ts:161:27)
    at start (/app/index.js:154:5)

A quick web search suggests that maybe the problem is that there are other routers around that are still running the network and that's causing some kind of conflict?

[bobobo1618]

Figured out the issue was the missing coordinator backup. Adding it got that error to go away.

[araemo]

I want to say thank you for this thread, and add some notes for anyone else who wants to try this:

• The script above (Successfully migrated from ZHA to Z2M without re-pairing #24478 (comment)) was very helpful, but as someone else noted, it didn't reverse the bytes of the extended pan id. And when I went and reversed them manually in configuration.yaml and restarted z2m, z2m helpfully noticed that I reversed them from the backup.. and ignored that I changed them.

[2024-12-27 22:52:38] warning: 	zh:adapter:zstack:manager: Extended PAN ID is reversed (expected=ad9838d3f5caa7d1, actual=d1a7caf5d33898ad)

So it took me a while to realize that was why it said this.. I originally assumed it was smarter than me, but none of my devices were talking right. Eventually I used the settings/advanced menu of z2m's front end to reverse the bytes and that fixed it (I assume deleting the coordinator_backup.json and changing the byte order in configuration.yaml, or simply doing it right before starting Z2M the first time would have worked too), and it started getting updates from my devices.

• Also related to that script, to get .storage/core.device_registry, create an HA backup, download it, and extract it.. inside is homeassistant.tar.gz, and inside that is data/.storage/core.device_registry (You can also get zigbee.db this way instead of the file editor addon)

• database.db population:

  1. https://github.com/Koenkk/zigbee-herdsman/blob/master/src/zspec/zcl/definition/manufacturerCode.ts I used this for manufacturer codes
  2. I converted the manufacturer code to decimal, ie: {"id":2,"type":"EndDevice","ieeeAddr":"0x00124b002c3aad24","nwkAddr":"4868","manufId":4897}
  3. I have a couple SONOFF devices that ZHA had detected as eWeLink... but after interviewing in Z2M, they show up as SONOFF correctly (though I did set them to sonoff's manufId in database.db, so I don't know if I cheated here)
  4. Not being familiar with database.db, I assumed it was JSON... but it isn't exactly. Each line is a full JSON object of a different device. So, it took me a few tries to realize that I didn't want to make the file a big JSON array or put commas after each device entry.
  5. And since my database.db already had an entry for my coordinator, don't include that entry from the ZHA zigbee.db
  6. to get the necesary info from ZHA zigbee.db, use this sqlite query:

SELECT devices_v13.ieee, devices_v13.nwk, attributes_cache_v13.value as manufacturer FROM devices_v13
join attributes_cache_v13 on devices_v13.ieee = attributes_cache_v13.ieee and attributes_cache_v13.attr_id = 4

[toine512]

Happy new year everyone.

I'm starting from scratch on the zigbee2mqtt side. Provided Z2M configuration is complete (Zigbee network...), does it hurt to start without a device list so that I get the base database.db file generated ?

[toine512]

I was wrong: if there is no "type": "Coordinator" entry it will try to create it and fail because the MAC address is already in database. Moreover if the coordinator entry is incomplete, it won't fill it. So starting with a blank db first is necessary to get the coordinator entry, then leave it out when copying from ZHA. Next, the manual interview has to be perfomed for each device. And it just works. Especially the Aqara stuff. (Except my ZLinky_TIC that keeps complaining but apparently that's expected.)
"Beauty!"

@thargy

Note you only need the IEEE and Network addresses. You can set the "type" to "Unknown". The interview will grab the correct type and everything else (including the Manufacturer ID).

"type": "Unknown" didn't work (devices ignored).

[toine512]

Hello @araemo.
You went all the way by filling the manufacturer ID in the database.db you made. Did that made your devices able to receive messages out of the box?
I didn't do it and Z2M ignores messages (devices marked as unsupported) until I run the interview.
I'm wondering if the effort of filling manufId makes a difference.

[araemo]

Everything still needed to be interviewed, but according to this comment #24478 (reply in thread), you only need the network addr and ieee addr.

Note you only need the IEEE and Network addresses. You can set the "type" to "Unknown". The interview will grab the correct type and everything else (including the Manufacturer ID).

So while I did this:

{
  "id": 4,
  "type": "EndDevice",
  "ieeeAddr": "0x00158d000ad9f8a8",
  "nwkAddr": "31299",
  "manufId": 4447
}

This should work the same:

{
  "id": 4,
  "type": "Unknown",
  "ieeeAddr": "0x00158d000ad9f8a8",
  "nwkAddr": "31299",
}

[thargy]

Yep, I redid 200+ devices using just type ‘Unknown’ with IEEE and network address (and sequential ID of course). Not sure what @toine512 ’s problem was?

A reinterview populated everything else.

[toine512]

This should work the same:

{
  "id": 4,
  "type": "Unknown",
  "ieeeAddr": "0x00158d000ad9f8a8",
  "nwkAddr": "31299",
}

That's what I did but with type: EndDevice. It works.

Yep, I redid 200+ devices using just type ‘Unknown’ with IEEE and network address (and sequential ID of course). Not sure what @toine512 ’s problem was?

With type Unknown Z2M was ignoring all entries (explicit message in the log), no device shown in the Web UI. With type EndDevice it's ok. Status is unsupported until interview is performed.

[toine512]

So here's what I did. Based on details in pretty much all previous posts. I performed data manipulation on the side for ease of use, config files are copied back and forth with the machine running zigbee2mqtt.

1. I plugged my ZBDongle-E in my PC and performed a coordinator backup in Open Coordinator Backup format using zigpy-cli:
   zigpy radio ezsp COM7 backup zigbee_backup.json
   The whole operation takes less than 3 min, including installing zigpy-cli with pip in a Python virtual environment.
   (Use zigpy radio ezsp <serial port> info for probing.)

2. I wrote a zigbee2mqtt configuration file from scratch following the manual. I used information from the coordinator backup to fill in all network configuration settings. When copying ext_pan_id, the order of bytes has to be reversed. The result looks like this:
   configuration.yaml

   permit_join: false
   mqtt:
       ...
   serial:
       ...
   frontend:
       ...
   advanced:
       pan_id: 0xb00b
       ext_pan_id: [0x08, 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01]
       channel: 26
       network_key: [0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10]
   devices: devices.yaml

   The devices list will be generated in a separate devices.yaml file.

3. I copied <HA data dir>/.storage/core.device_registry and <HA data dir>/zigbee.db files from Home Assistant.

4. I used the following Python script to generate (devices.yaml) the MAC address to friendly name association from core.device_registry file:
   devicelist.py

   from pathlib import Path
   import json
   
   with Path("core.device_registry").open("rt", encoding="utf-8", errors="strict") as fd_in:
   	ha_devices = json.load(fd_in)["data"]["devices"]
   
   with Path("devices.yaml").open("wt", encoding="utf-8", errors="strict", newline="\n") as fd_out:
   	for dev in ha_devices:
   		try:
   			if dev["identifiers"][0][0] != "zha":
   				continue
   		except IndexError:
   			continue
   
   		macaddr = dev["identifiers"][0][1]
   		macaddr = macaddr.replace(":", "")
   		name = dev["name_by_user"] if dev["name_by_user"] else dev["name"]
   		name = name.replace("'", "''")
   
   		print(f"'0x{macaddr}':", f"    friendly_name: '{name}'", sep="\n", file=fd_out)

   Result

   devices.yaml

   '0xe0798dfffe8b8bc5':
       friendly_name: 'ZBDongle-E'
   '0x00158d00083046e3':
       friendly_name: 'Sonde Aqara Salon'
   '0x00158d00083044cb':
       friendly_name: 'Sonde Aqara Chambre'
   '0x00124b0024c0876a':
       friendly_name: 'Radiateur Chambre'
   '0x00124b0024c5dcbb':
       friendly_name: 'Radiateur Salon'
   '0x00158d0005be9a07':
       friendly_name: 'LiXee ZLinky_TIC'
   '0x04cf8cdf3c7d2144':
       friendly_name: 'LUMI lumi.sen_ill.mgl01'
   '0x00158d0008301503':
       friendly_name: 'Sonde Aqara Extérieur'
   '0x00158d000758eadb':
       friendly_name: 'LUMI lumi.sensor_motion.aq2'
   '0x04cf8cdf3c8dc11a':
       friendly_name: 'LUMI lumi.sen_ill.agl01'
   '0x9035eafffe7e932c':
       friendly_name: 'IKEA of Sweden TRADFRI bulb E14 CWS 470lm'
   '0x00158d000879bb4f':
       friendly_name: 'LUMI lumi.remote.b286acn02'
   '0x00124b002b49a1fc':
       friendly_name: 'SONOFF S26R2ZB'

5. I copied configuration.yaml and devices.yaml to <Z2M directory>/data/ in a fresh installation of zigbee2mqtt and started it (with the coordinator attached). Now is time to sort out any issue with your base configuration of zigbee2mqtt. Once a complete startup has been performed without error and communication with the coordinator is confirmed, stop zigbee2mqtt and proceed to next step.

6. zigbee2mqtt has now generated a database.db file at the root of its directory structure which contains one line with "id":1 and "type":"Coordinator" that holds information about the coordinator. I copied this file and passed it to the following script that transfers Zigbee network devices entries from ZHA's SQLite3 database zigbee.db to zigbee-herdsman's JSON database.db:
   dbimport.py

   import sqlite3
   from pathlib import Path
   
   con = sqlite3.connect("zigbee.db")
   cur = con.cursor()
   res = cur.execute("SELECT ieee, nwk FROM devices_v12 WHERE nwk != 0") # Omit the coordinator already in database.db
   mapping = res.fetchall()
   con.close()
   
   with Path("database2.db").open("at", encoding="utf-8", errors="strict", newline="\n") as fd:
   	# It is assumed that the coordinator is the only line in database.db, with id: 1
   	for id, (mac_addr, net_addr) in enumerate(mapping, start=2):
   		mac_addr = mac_addr.replace(":", "")
   		fd.write(f'\n{{"id":{id},"type":"EndDevice","ieeeAddr":"0x{mac_addr}","nwkAddr":{net_addr}}}')

   "id" is incremented starting at 2, "type" is set to "EndDevice", "ieeeAddr" and "nwkAddr" are copied from ZHA.
   The zigbee.db table name is "devices_v13" for latest Home Assistant.

   Result

   database.db

   {"id":1,"type":"Coordinator","ieeeAddr":"0xe0798dfffe8b8bc5","nwkAddr":0,"manufId":4169,"epList":[1,242],  ...
   {"id":2,"type":"EndDevice","ieeeAddr":"0x00158d00083046e3","nwkAddr":22456}
   {"id":3,"type":"EndDevice","ieeeAddr":"0x00158d00083044cb","nwkAddr":61473}
   {"id":4,"type":"EndDevice","ieeeAddr":"0x00124b0024c0876a","nwkAddr":10039}
   {"id":5,"type":"EndDevice","ieeeAddr":"0x00124b0024c5dcbb","nwkAddr":58880}
   {"id":6,"type":"EndDevice","ieeeAddr":"0x00158d0005be9a07","nwkAddr":49129}
   {"id":7,"type":"EndDevice","ieeeAddr":"0x04cf8cdf3c7d2144","nwkAddr":28681}
   {"id":8,"type":"EndDevice","ieeeAddr":"0x00158d0008301503","nwkAddr":49679}
   {"id":9,"type":"EndDevice","ieeeAddr":"0x00158d000758eadb","nwkAddr":50663}
   {"id":10,"type":"EndDevice","ieeeAddr":"0x04cf8cdf3c8dc11a","nwkAddr":62978}
   {"id":11,"type":"EndDevice","ieeeAddr":"0x9035eafffe7e932c","nwkAddr":2883}
   {"id":12,"type":"EndDevice","ieeeAddr":"0x00158d000879bb4f","nwkAddr":62589}
   {"id":13,"type":"EndDevice","ieeeAddr":"0x00124b002b49a1fc","nwkAddr":47871}

   New lines are appended to database.db.

7. Then I copied the new database.db to its original location and started zigbee2mqtt. All your devices should appear.
   Since zigbee2mqtt has no information about devices yet, they all are in unsupported state.
   I then requested interview for each device. All details are retrieved from the device and it starts operating in Z2M.

   The usual wisdom applies: first interview mains-powered devices that you know are routers, then all mains powered devices, then battery-powered ones. You will have to go and wake up each battery-powered device.

Everything works.

[rjDipcord]

I think I followed this writeup to a T as you have written it, but I get stuck at step 5. Seems Z2M is upset about migrating.

[2025-01-04 00:47:10] error: 	z2m: Error while starting zigbee-herdsman
[2025-01-04 00:47:10] error: 	z2m: Failed to start zigbee-herdsman
[2025-01-04 00:47:10] error: 	z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html for possible solutions
[2025-01-04 00:47:10] error: 	z2m: Exiting...
[2025-01-04 00:47:10] error: 	z2m: Error: network commissioning timed out - most likely network with the same panId or extendedPanId already exists nearby (Error: AREQ - ZDO - stateChangeInd after 60000ms

Other solutions to this problem only suggest to generate new keys. But since I am trying to keep my network intact, I would like to avoid that as much as possible!

[toine512]

Did you reverse the order of bytes of ext_pan_id and not of pan_id and network_key?

As long as it doesn't work right, make sure to remove data/coordinator_backup.json if it gets created upon Z2M exit.

[rjDipcord]

Did you reverse the order of bytes of ext_pan_id and not of pan_id and network_key?

Yep, exactly.

I am not seeing coordinator_backup.json get created. However I am unsure I am looking in the right place. Are you using /data in reference to the Z2M root config folder in HA's /config directory? That is, data/coordinator_backup.json would live in the same directory as configuration.yaml and devices.yaml?

[toine512]

That is, data/coordinator_backup.json would live in the same directory as configuration.yaml and devices.yaml?

Yes.

[Zzo911]

Dear all,
Many thanks for this helpfull thread.
I have a question regarding the entity names of the devices migrated from ZHA to Z2M.
In ZHA, I have for example sensor.device_name_summation_delivered (sum in kWh).
Once migrated to Z2H, the entity is called sensor.device_name_energy.
How can I rename it from _energy to _summation_delivered in home assistant and get back the historical data?
Thanks

[Zzo911]

OK, I'll look at this device type.
I just saw Z2M updated to v2.0.0 today, I hope the process to migrate is still the same.

[toine512]

Thank you for pointing it out. Reading #24198, I found no changes to network parameters or the very simple device list that is generated. So it should be ok. I hope @rjDipcord's issue doesn't come from this.

There is a massive configuration.yaml reorganization though.
(And my previous illuminance sensor mixup wouldn't happen with the new version.)

[Zzo911]

Not good... I have this error when starting Z2M v2 after the update migrate the configuration.yaml.

[15:24:57] INFO: Preparing to start...
[15:24:57] INFO: Socat not enabled
[15:24:57] INFO: Starting Zigbee2MQTT...
Starting Zigbee2MQTT without watchdog.
Migration notes written in /config/zigbee2mqtt/migration-1-to-2.log
Migration notes written in /config/zigbee2mqtt/migration-2-to-3.log
Migration notes written in /config/zigbee2mqtt/migration-3-to-4.log
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
            READ THIS CAREFULLY
Refusing to start because configuration is not valid, found the following errors:
- devices must match pattern "^0x[\d\w]{16}$"
- devices property name must be valid
- devices must match pattern "^0x[\d\w]{16}$"
- devices property name must be valid
If you don't know how to solve this, read https://www.zigbee2mqtt.io/guide/configuration
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

In configuration.yaml I have my devices as:

devices:
  '0xa4c1380003b9de58':
    friendly_name: LED panel

[Zzo911]

All good now, for an unknown reason the last 2 devices where missing the leading "0x" once migrated.
Continuing...

[Zzo911]

@toine512 So I was able to get the sensor history back.
Here is what I did, probably not the best approach but working.

I have a device called "congelateur" which is this one https://www.zigbee2mqtt.io/devices/TS011F_plug_1.html#tuya-ts011f_plug_1
ZHA old entity name : sensor.congelateur_consommation
Z2M new entity name : sensor.congelateur_energy

1 . Delete the entity sensor.congelateur_consommation, home assistant did not purge the data
2. Rename sensor.congelateur_energy to sensor.congelateur_consommation

Now, if I want to keep Z2M naming convention for future devices like this one:

1. Go to the development tools and delete then entity sensor.congelateur_energy
2. Rename sensor.congelateur_consommation to sensor.congelateur_energy, the history will move with the new id
3. Fix any automation/script that may use sensor.congelateur_consommation

[rdehuyss]

Would someone be able to write a (detailed) howto on how to do this? I've started of with ZHA but feel this is a mistake:

• all my end devices deliver wrong battery info
• the TRVs I bought of which I thought there was a quirk now have a different fingerprint and are not supported, ... .

You would have everlasting ❤️ 🙏 .

[atallo]

Toine512 made an almost, almost perfect manual.

Thanks Antoni! I managed to migrate my network without re-pairing (And thank you XX seidtgeist your idea of ​​starting this post)
#24478 (comment)

[teal-bauer]

I believe this should work?

Using the Home Assistant Terminal and Studio Code Server add-ons (for easier editing/access),

in Terminal run:

$ tail -n 1 config/zigbee2mqtt/database.db
# note the highest ID value, replace below where it says "+ 2"
$ sqlite3 config/zigbee.db           
sqlite> select CONCAT('{"id": ', (ROW_NUMBER() OVER (ORDER BY d.ieee)) + 2, ', "type": "EndDevice", "ieeeAddr": "0x', REPLACE(d.ieee, ':', ''), '", "nwkAddr": ', d.nwk, '}') FROM devices d JOIN endpoints e ON d.ieee = e.ieee ;

This will give you a list like

{"id": 8, "type": "EndDevice", "ieeeAddr": "0x00178801023dd160", "nwkAddr": 23188}

That you should be able to paste into the zigbee2mqtt/database.db.
Then start Z2M, and it ought to just work?
This doesn't give you friendly names - I'll leave that as an exercise for the reader... :D

[teal-bauer]

Also note that you still have to manually copy, reformat and potentially reverse PAN, ePAN, network key etc. those are visible in the ZHA config or the zigbee.db networks config.

[rdehuyss]

I have my first device up & running with Z2M but after reading this thread and owning the SLZB-06M, I'm quite scared to be honest to make the move. ZHA has been running stable here but it does not support all the devices that I have. Anybody else here has the SLZB-06M?