Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.220347487s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 6.22036975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.746825462s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.746854836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.775479663s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:13:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.775510231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.743192673s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:12:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.743256955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.60496986s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:12:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.605001029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.219103217s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:11:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 6.219122703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.843317094s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:11:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.843353833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.893145196s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:10:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.893172628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.686212273s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:09:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.686248472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.674885794s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:08:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.67491468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.699379507s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:07:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.699411519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.654185368s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:06:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.65421262s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.690880336s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:05:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.690902097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.680339208s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:05:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.680402608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.757393926s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.75742272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.900458758s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.900610908s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.758546633s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:03:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.758651641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.846745141s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:03:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.846773435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.664996424s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:02:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.665028715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.886089277s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.886119805s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.695858645s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:01:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.695877962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.742248598s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 07:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.742279378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.625992714s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:59:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.62602269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.647849581s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:58:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.647873578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.65735441s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:58:40 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.65738515s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.7848552s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:58:06 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.784880127s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.737172391s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:58:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.737204501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.797103385s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.797137281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.785833203s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:56:57 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.785861627s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.746978779s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:56:30 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.747015671s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.629538855s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:56:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.629570365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.815509963s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:56:02 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.815544408s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.810071275s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:55:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.810101614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.852263209s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:55:37 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.852292824s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.693115641s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:54:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.693145319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.828715988s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:54:24 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.828746337s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.765932249s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:54:08 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.765974589s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.897906191s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:53:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.897928963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.813370018s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:53:49 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.813403069s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.588167486s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:53:25 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.588207122s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.656336074s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:53:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.656413091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.664063478s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:52:38 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.664085251s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.863755453s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:52:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.863781122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.508319134s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:51:50 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.508346615s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.737902902s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:51:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.737936187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.871923291s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:50:44 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.871950914s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.687168847s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:50:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.687188414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.192718212s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:50:06 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.192751997s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.705959385s trace: https://jaeger.dev.storj.io/trace/56c7a5b54e906450 FAIL (2025-11-25 06:49:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 25 06:32:59.722153851 (382h28m58.273624312s) [upload=336846]: Starting upload Nov 25 06:32:59.722267227 (382h28m58.273737698s) [upload=336846]: Done starting upload Nov 25 06:32:59.722340045 (382h28m58.273810516s) [upload=336846]: Uploading segments... Nov 25 06:32:59.722575253 (382h28m58.274045724s) [upload=336846]: Got next segment. Inline: false Nov 25 06:32:59.722619888 (382h28m58.274090349s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 06:32:59.722626211 (382h28m58.274096672s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 06:33:00.818993928 (382h28m59.370464379s) [seg_pos={0 0}, upload=336846]: Begin upload segment... Nov 25 06:33:00.819053422 (382h28m59.370523943s) [seg_pos={0 0}, upload=336846]: Done begin upload segment. Nov 25 06:33:00.819110190 (382h28m59.370580651s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819148944 (382h28m59.370619405s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.819873394 (382h28m59.371343855s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820280608 (382h28m59.371751069s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.820709486 (382h28m59.372179947s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.821138893 (382h28m59.372609354s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.822751804 (382h28m59.374222265s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.824666450 (382h28m59.376136901s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.827883436 (382h28m59.379353897s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.832546405 (382h28m59.384016866s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.836202727 (382h28m59.387673188s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.838759827 (382h28m59.390230288s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.840211380 (382h28m59.391681841s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.841702119 (382h28m59.39317258s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.845441981 (382h28m59.396912452s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:00.847545015 (382h28m59.399015476s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Uploading piece... Nov 25 06:33:02.648887708 (382h29m1.200358179s) [upload=336846]: Next returned nil segment Nov 25 06:33:02.650912935 (382h29m1.202383426s) [upload=336846]: Waiting for error group managing segments... Nov 25 06:33:02.986613255 (382h29m1.538083716s) [noise=true , address=77.90.31.11:4554 , piece_id=U2L3CIZRVZ5GURKXZB47RM6WNS4HPBF27I42LTF3RMIB5CTR4MUQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.033304084 (382h29m1.584774555s) [noise=true , address=77.90.31.10:4518 , piece_id=4FTNQ7TAMDUXOU2EIQTE4WG36KXM6R5PCDVOHKNIRVNJMZ2GTQRA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.035864800 (382h29m1.587335261s) [noise=true , address=77.90.31.19:4841 , piece_id=6PDKD3JT34RNBS673OGAQGUMSF6ZHDJXYKMXZCQNH27GIBWZD72Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.071371212 (382h29m1.622841683s) [noise=true , address=145.79.227.21:4473 , piece_id=DGSNE56UBKSUHUNJ5RMBJD6LVBPJZY5KJF272LVQWTRRMZLUEVHA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.092380708 (382h29m1.643851169s) [noise=true , address=145.79.227.9:4366 , piece_id=FHI7FO2BUECKBOSMSC5JG7VFG2YNZQKC2E5JBBEZQCODSUCS5JKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:03.222496424 (382h29m1.773966885s) [noise=true , address=145.79.227.3:4150 , piece_id=UKZOYGH557QBMQPUCVVJ2SUQWWIRABKMPUFERVI2V5MQPUN3UJOQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.204885482 (382h29m2.756355943s) [noise=true , address=145.79.227.4:4186 , piece_id=YVNR4QCAO5PCOCE7VXIC2MK6QN5Y7HCP33WGYYKTD47KRCPY6YKA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.209566787 (382h29m2.761037248s) [noise=true , address=145.79.227.13:4802 , piece_id=JL6BMDM53Q3THMAA333C4VDXSLARLD743JBYJOJWUMHKNE2MXLVA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:04.562470498 (382h29m3.113940959s) [noise=true , address=145.79.227.15:4874 , piece_id=JI3CB6A5KZL6IDXDPNPK6X6Z3CXVC3XUC4SS2KUNHLXY2O5KEQ4Q, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:05.630708178 (382h29m4.182178639s) [noise=true , address=145.79.227.22:4510 , piece_id=BISZBJZKPG6KHVOCO7WY2LL4PVZ2HU2TO54GPUJCCQNX7Y63PEBA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.433366150 (382h29m4.984836601s) [noise=true , address=145.79.227.17:4971 , piece_id=IZU6PNH4KCJYJNVPBDIGGJNA3FIGRYRVRLDIXPPIFL6ZWCYAGQTQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:06.682522955 (382h29m5.233993416s) [noise=true , address=145.79.227.34:4905 , piece_id=S3IYBEAQNG4DVKL6ME7AYFZ75M4FP2MBSKW4EOQJYRGXOY4U4MGA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170551901 (382h29m5.722022372s) [noise=true , address=145.79.227.25:4618 , piece_id=FF2PWN4YDYVT2SGMDGCNKQCY3IF3S7IYF72MNREFNR7IGV27G4PA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: <nil> Nov 25 06:33:07.170579023 (382h29m5.722049484s) [seg_pos={0 0}, upload=336846]: Segment reached optimal threshold of 13 pieces. Nov 25 06:33:07.170809632 (382h29m5.722280093s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38062->145.79.227.42:4123: use of closed network connection Nov 25 06:33:07.170818960 (382h29m5.722289421s) [noise=true , address=145.79.227.42:4123 , piece_id=YCYEJFPW5X6JUDSXJGXHKNJRA664TFZSITEMJKP4G3ED2UHPRFFA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.170878103 (382h29m5.722348564s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.170887400 (382h29m5.722357861s) [noise=true , address=145.79.227.30:4762 , piece_id=C4L36VQZJSIABZGLN5TVGNP5L4SB4YFGAZKXXNKQ44MIZETIKEPA, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171001848 (382h29m5.722472309s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Nov 25 06:33:07.171007899 (382h29m5.72247836s) [noise=true , address=145.79.227.5:4222 , piece_id=73NEU2HYT2N7QBCW7RNV2KZX4U3VLZBB6EZG3OHKNRLLUVR6BJNQ, seg_pos={0 0}, upload=336846]: Long tail cancellation due to optimal threshold Nov 25 06:33:07.171024341 (382h29m5.722494802s) [seg_pos={0 0}, upload=336846]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 25 06:33:07.171140943 (382h29m5.722611404s) [upload=336846]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 25 06:33:07.171147926 (382h29m5.722618387s) [upload=336846]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 25 06:33:11.966852121 (382h29m10.518322592s) [upload=336846]: Done uploading segments... Nov 25 06:33:11.966885565 (382h29m10.518356016s) [upload=336846]: Upload finished. err: <nil> Duration: 5.706037985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.842132613s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:49:33 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.842150187s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)