Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X.] duration: 5.963214074s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:04:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 5.963240074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X.....X.........X.....X.................X.....X.] duration: 3.645243378s trace: https://jaeger.dev.storj.io/trace/5d8ddb87b498960b FAIL (2025-12-27 07:04:40 UTC) orbiter warning: context deadline exceeded Duration: 3.645269939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [......XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X] duration: 30.003417631s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 30.003447268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...................................X......X..] duration: 55.603626875s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 55.603727788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X.....X.........X.....X.................X.....X] duration: 5.001576145s trace: https://jaeger.dev.storj.io/trace/5d8ddb87b498960b FAIL (2025-12-27 07:03:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001601104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.] duration: 8.293333863s trace: https://jaeger.dev.storj.io/trace/36a64374731ba559 FAIL (2025-12-27 07:03:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ) context deadline exceeded ======LOGS====== 07:02:19.713260012 (519h33m19.584343057s) [upload=455816]: Starting upload Dec 27 07:02:19.713370101 (519h33m19.584453136s) [upload=455816]: Done starting upload Dec 27 07:02:19.713494889 (519h33m19.584577944s) [upload=455816]: Uploading segments... Dec 27 07:02:19.713686253 (519h33m19.584769288s) [upload=455816]: Got next segment. Inline: false Dec 27 07:02:19.713735908 (519h33m19.584818963s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:02:19.713742330 (519h33m19.584825375s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:02:19.958209063 (519h33m19.829292108s) [seg_pos={0 0}, upload=455816]: Begin upload segment... Dec 27 07:02:19.958254991 (519h33m19.829338037s) [seg_pos={0 0}, upload=455816]: Done begin upload segment. Dec 27 07:02:19.958285078 (519h33m19.829368113s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958323882 (519h33m19.829406918s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958893216 (519h33m19.829976271s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959334206 (519h33m19.830417252s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959769825 (519h33m19.830852881s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960182051 (519h33m19.831265086s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960530935 (519h33m19.83161405s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960890850 (519h33m19.831973885s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961261426 (519h33m19.832344461s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961644986 (519h33m19.832728041s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962060037 (519h33m19.833143082s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962414752 (519h33m19.833497787s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962694575 (519h33m19.83377762s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963023461 (519h33m19.834106507s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963280591 (519h33m19.834363626s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963538572 (519h33m19.834621617s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:21.584837547 (519h33m21.455920592s) [upload=455816]: Next returned nil segment Dec 27 07:02:21.584854599 (519h33m21.455937644s) [upload=455816]: Waiting for error group managing segments... Dec 27 07:02:21.960640250 (519h33m21.831723285s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.961296279 (519h33m21.832379314s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.963953889 (519h33m21.835036924s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.969000689 (519h33m21.840083724s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.973197301 (519h33m21.844280346s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.976976758 (519h33m21.848059793s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989097227 (519h33m21.860180272s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989403661 (519h33m21.860486706s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989946515 (519h33m21.86102956s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.990611040 (519h33m21.861694085s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.001852716 (519h33m21.872935761s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.246244286 (519h33m22.117327321s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266438771 (519h33m22.137521806s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266457596 (519h33m22.137540631s) [seg_pos={0 0}, upload=455816]: Segment reached optimal threshold of 13 pieces. Dec 27 07:02:22.266662517 (519h33m22.137745552s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection Dec 27 07:02:22.266668267 (519h33m22.137751313s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266863299 (519h33m22.137946344s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266901853 (519h33m22.137984898s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266906722 (519h33m22.137989757s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266913454 (519h33m22.137996489s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.267006572 (519h33m22.138089607s) [seg_pos={0 0}, upload=455816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:02:22.267090912 (519h33m22.138173957s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:02:22.267095682 (519h33m22.138178717s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:02:22.554710707 (519h33m22.425793753s) [upload=455816]: Done uploading segments... Dec 27 07:02:22.554737639 (519h33m22.425820684s) [upload=455816]: Upload finished. err: <nil> Duration: 8.293369692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...................................X......X.] duration: 43.382065533s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:03:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 43.3820903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X.....X.........X.....X.................X.....] duration: 3.654062603s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:02:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654114422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX] duration: 30.00278291s trace: https://jaeger.dev.storj.io/trace/36a64374731ba559 FAIL (2025-12-27 07:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ) context deadline exceeded ======LOGS====== 07:02:19.713260012 (519h33m19.584343057s) [upload=455816]: Starting upload Dec 27 07:02:19.713370101 (519h33m19.584453136s) [upload=455816]: Done starting upload Dec 27 07:02:19.713494889 (519h33m19.584577944s) [upload=455816]: Uploading segments... Dec 27 07:02:19.713686253 (519h33m19.584769288s) [upload=455816]: Got next segment. Inline: false Dec 27 07:02:19.713735908 (519h33m19.584818963s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:02:19.713742330 (519h33m19.584825375s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:02:19.958209063 (519h33m19.829292108s) [seg_pos={0 0}, upload=455816]: Begin upload segment... Dec 27 07:02:19.958254991 (519h33m19.829338037s) [seg_pos={0 0}, upload=455816]: Done begin upload segment. Dec 27 07:02:19.958285078 (519h33m19.829368113s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958323882 (519h33m19.829406918s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958893216 (519h33m19.829976271s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959334206 (519h33m19.830417252s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959769825 (519h33m19.830852881s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960182051 (519h33m19.831265086s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960530935 (519h33m19.83161405s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960890850 (519h33m19.831973885s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961261426 (519h33m19.832344461s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961644986 (519h33m19.832728041s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962060037 (519h33m19.833143082s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962414752 (519h33m19.833497787s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962694575 (519h33m19.83377762s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963023461 (519h33m19.834106507s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963280591 (519h33m19.834363626s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963538572 (519h33m19.834621617s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:21.584837547 (519h33m21.455920592s) [upload=455816]: Next returned nil segment Dec 27 07:02:21.584854599 (519h33m21.455937644s) [upload=455816]: Waiting for error group managing segments... Dec 27 07:02:21.960640250 (519h33m21.831723285s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.961296279 (519h33m21.832379314s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.963953889 (519h33m21.835036924s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.969000689 (519h33m21.840083724s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.973197301 (519h33m21.844280346s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.976976758 (519h33m21.848059793s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989097227 (519h33m21.860180272s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989403661 (519h33m21.860486706s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989946515 (519h33m21.86102956s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.990611040 (519h33m21.861694085s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.001852716 (519h33m21.872935761s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.246244286 (519h33m22.117327321s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266438771 (519h33m22.137521806s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266457596 (519h33m22.137540631s) [seg_pos={0 0}, upload=455816]: Segment reached optimal threshold of 13 pieces. Dec 27 07:02:22.266662517 (519h33m22.137745552s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection Dec 27 07:02:22.266668267 (519h33m22.137751313s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266863299 (519h33m22.137946344s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266901853 (519h33m22.137984898s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266906722 (519h33m22.137989757s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266913454 (519h33m22.137996489s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.267006572 (519h33m22.138089607s) [seg_pos={0 0}, upload=455816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:02:22.267090912 (519h33m22.138173957s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:02:22.267095682 (519h33m22.138178717s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:02:22.554710707 (519h33m22.425793753s) [upload=455816]: Done uploading segments... Dec 27 07:02:22.554737639 (519h33m22.425820684s) [upload=455816]: Upload finished. err: <nil> Duration: 30.002808429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...................................X......X] duration: 1m0.003418863s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:02:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 1m0.003436557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X.....X.........X.....X.................X....] duration: 3.734486175s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:02:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.734538895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XX] duration: 30.004704506s trace: https://jaeger.dev.storj.io/trace/287bb601b51eaac FAIL (2025-12-27 07:01:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 07:01:19.259902587 (519h32m19.130985632s) [upload=455803]: Starting upload Dec 27 07:01:19.259958274 (519h32m19.131041309s) [upload=455803]: Done starting upload Dec 27 07:01:19.260173103 (519h32m19.131256148s) [upload=455803]: Uploading segments... Dec 27 07:01:19.260257703 (519h32m19.131340738s) [upload=455803]: Got next segment. Inline: false Dec 27 07:01:19.260288272 (519h32m19.131371307s) [upload=455803]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:01:19.260292349 (519h32m19.131375384s) [upload=455803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:01:19.513066440 (519h32m19.384149485s) [seg_pos={0 0}, upload=455803]: Begin upload segment... Dec 27 07:01:19.513120362 (519h32m19.384203397s) [seg_pos={0 0}, upload=455803]: Done begin upload segment. Dec 27 07:01:19.513136554 (519h32m19.384219589s) [noise=true , address=145.79.227.47:4265 , piece_id=LG44OFTAQU2GWGFTIEWFRS5HKYD3GFZ6USTKT4B4HPRQXDXL64PA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.513194684 (519h32m19.384277719s) [noise=true , address=145.79.227.35:4482 , piece_id=Z3KC5QOZ3H25DPWWNCRCTOKY46GPN4KLEFBMS4XLMM7PE2N6HQ7Q, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.513950603 (519h32m19.385033649s) [noise=true , address=77.90.31.18:4808 , piece_id=DTO6GBRYT2OFLLMGAWBY5EUJ2SFDATNYU2FPJRVPDMVD334BGC6A, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.514250074 (519h32m19.385333109s) [noise=true , address=145.79.227.29:4943 , piece_id=O7GQ4GSRF7VD4CJWCQXB4BTYHPDT77YYSNLYPFRGP2DKPJVNTRUA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.514573660 (519h32m19.385656695s) [noise=true , address=145.79.227.5:4221 , piece_id=5ERYTHGIAVX6G2RQXHGN2NARS47E4VYATEOCUXQLLPOKB6EK3Q6A, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.514852912 (519h32m19.385935947s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRZNVGWCTS33W45BCKB2E2U55HLP6RAERARMLG7OBRXYJPTYBWQ, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.515123427 (519h32m19.386206462s) [noise=true , address=145.79.227.51:4409 , piece_id=NPW7PTV3OUQCSFVU6OVWBGZQNM3TBZN7NKEO5G46QH55O4JEKALA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.515393031 (519h32m19.386476066s) [noise=true , address=145.79.227.38:4977 , piece_id=QNDAQVLKEXT2KOZOJS6UHV3DBFYQ5LSFR2HJJ4PPWPHTTQDM2V4Q, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.515910987 (519h32m19.386994032s) [noise=true , address=77.90.31.11:4554 , piece_id=5DW2GCZCBHLV2B2V24NGXI7P5RESP3ELE2OKCMG6JCITCLHUWWEA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.516846598 (519h32m19.387929643s) [noise=true , address=145.79.227.28:4727 , piece_id=LAJ5TPN33HCGQWEGUGCEQXN435HLE36OT4QIU5YKTX4TGCIVKJQQ, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.517128805 (519h32m19.388211841s) [noise=true , address=145.79.227.36:4519 , piece_id=ABCM3AWPVC3PGQSZYIP2VZRV7SCU7NXKZ5IH6MSTARENCVIMPBRA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.518056111 (519h32m19.389139146s) [noise=true , address=145.79.227.12:4765 , piece_id=33PMSKVSDLMZXOHMQ6IIO62YJRH7JKHCD44252NQLGXYEG7QGOMA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.518393394 (519h32m19.389476429s) [noise=true , address=45.91.116.11:4118 , piece_id=W52N7H4U3YUSTINBSG5FXHT7ZKJ2X2W77FEOB4H7UR5DWAE3VOKA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.519148031 (519h32m19.390231066s) [noise=true , address=77.90.31.17:4770 , piece_id=YTUPGZMZUZNU66QLUO7CMSO3HCDXWV5HDICX4UX7LMBVJWTV4Y3A, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.520245640 (519h32m19.391328676s) [noise=true , address=145.79.227.52:4445 , piece_id=JLILBLAGC36AHPA43EQL3CXTERHSYSOCN7MJ2LC5U2DFHOBZ72SA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.520710816 (519h32m19.391793861s) [noise=true , address=145.79.227.6:4259 , piece_id=RLDWE26JG6IJZ6OUBJOD2WAW7GKOHCOTT4334L5RP3WGCLZXMV5Q, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:21.141571877 (519h32m21.012654912s) [upload=455803]: Next returned nil segment Dec 27 07:01:21.141589611 (519h32m21.012672656s) [upload=455803]: Waiting for error group managing segments... Dec 27 07:01:21.525850640 (519h32m21.396933675s) [noise=true , address=145.79.227.36:4519 , piece_id=ABCM3AWPVC3PGQSZYIP2VZRV7SCU7NXKZ5IH6MSTARENCVIMPBRA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.528154137 (519h32m21.399237172s) [noise=true , address=145.79.227.35:4482 , piece_id=Z3KC5QOZ3H25DPWWNCRCTOKY46GPN4KLEFBMS4XLMM7PE2N6HQ7Q, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.528640874 (519h32m21.399723909s) [noise=true , address=145.79.227.38:4977 , piece_id=QNDAQVLKEXT2KOZOJS6UHV3DBFYQ5LSFR2HJJ4PPWPHTTQDM2V4Q, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.529180741 (519h32m21.400263776s) [noise=true , address=145.79.227.5:4221 , piece_id=5ERYTHGIAVX6G2RQXHGN2NARS47E4VYATEOCUXQLLPOKB6EK3Q6A, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.530809713 (519h32m21.401892759s) [noise=true , address=145.79.227.12:4765 , piece_id=33PMSKVSDLMZXOHMQ6IIO62YJRH7JKHCD44252NQLGXYEG7QGOMA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.531873480 (519h32m21.402956525s) [noise=true , address=145.79.227.51:4409 , piece_id=NPW7PTV3OUQCSFVU6OVWBGZQNM3TBZN7NKEO5G46QH55O4JEKALA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.541565735 (519h32m21.41264879s) [noise=true , address=145.79.227.52:4445 , piece_id=JLILBLAGC36AHPA43EQL3CXTERHSYSOCN7MJ2LC5U2DFHOBZ72SA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.543842721 (519h32m21.414925756s) [noise=true , address=145.79.227.28:4727 , piece_id=LAJ5TPN33HCGQWEGUGCEQXN435HLE36OT4QIU5YKTX4TGCIVKJQQ, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.548417933 (519h32m21.419500978s) [noise=true , address=145.79.227.29:4943 , piece_id=O7GQ4GSRF7VD4CJWCQXB4BTYHPDT77YYSNLYPFRGP2DKPJVNTRUA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.551008146 (519h32m21.422091191s) [noise=true , address=145.79.227.6:4259 , piece_id=RLDWE26JG6IJZ6OUBJOD2WAW7GKOHCOTT4334L5RP3WGCLZXMV5Q, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.553580584 (519h32m21.424663619s) [noise=true , address=145.79.227.47:4265 , piece_id=LG44OFTAQU2GWGFTIEWFRS5HKYD3GFZ6USTKT4B4HPRQXDXL64PA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.706087037 (519h32m21.577170082s) [noise=true , address=77.90.31.18:4808 , piece_id=DTO6GBRYT2OFLLMGAWBY5EUJ2SFDATNYU2FPJRVPDMVD334BGC6A, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.985193130 (519h32m21.856276195s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRZNVGWCTS33W45BCKB2E2U55HLP6RAERARMLG7OBRXYJPTYBWQ, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.985216946 (519h32m21.856299991s) [seg_pos={0 0}, upload=455803]: Segment reached optimal threshold of 13 pieces. Dec 27 07:01:21.985542405 (519h32m21.85662544s) [noise=true , address=45.91.116.11:4118 , piece_id=W52N7H4U3YUSTINBSG5FXHT7ZKJ2X2W77FEOB4H7UR5DWAE3VOKA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:21.985549979 (519h32m21.856633024s) [noise=true , address=45.91.116.11:4118 , piece_id=W52N7H4U3YUSTINBSG5FXHT7ZKJ2X2W77FEOB4H7UR5DWAE3VOKA, seg_pos={0 0}, upload=455803]: Long tail cancellation due to optimal threshold Dec 27 07:01:21.985782893 (519h32m21.856865928s) [noise=true , address=77.90.31.17:4770 , piece_id=YTUPGZMZUZNU66QLUO7CMSO3HCDXWV5HDICX4UX7LMBVJWTV4Y3A, seg_pos={0 0}, upload=455803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:21.985836675 (519h32m21.85691972s) [noise=true , address=77.90.31.17:4770 , piece_id=YTUPGZMZUZNU66QLUO7CMSO3HCDXWV5HDICX4UX7LMBVJWTV4Y3A, seg_pos={0 0}, upload=455803]: Long tail cancellation due to optimal threshold Dec 27 07:01:21.985938649 (519h32m21.857021684s) [noise=true , address=77.90.31.11:4554 , piece_id=5DW2GCZCBHLV2B2V24NGXI7P5RESP3ELE2OKCMG6JCITCLHUWWEA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:59798->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:59798->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59798->77.90.31.11:4554: use of closed network connection Dec 27 07:01:21.985944221 (519h32m21.857027266s) [noise=true , address=77.90.31.11:4554 , piece_id=5DW2GCZCBHLV2B2V24NGXI7P5RESP3ELE2OKCMG6JCITCLHUWWEA, seg_pos={0 0}, upload=455803]: Long tail cancellation due to optimal threshold Dec 27 07:01:21.985959680 (519h32m21.857042725s) [seg_pos={0 0}, upload=455803]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:01:21.986019263 (519h32m21.857102298s) [upload=455803]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:21.986023702 (519h32m21.857106747s) [upload=455803]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:22.287013235 (519h32m22.15809628s) [upload=455803]: Done uploading segments... Dec 27 07:01:22.287034145 (519h32m22.15811719s) [upload=455803]: Upload finished. err: <nil> Duration: 30.004717641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X.....X.........X.....X.................X...] duration: 3.48726263s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487295613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..X] duration: 30.002373838s trace: https://jaeger.dev.storj.io/trace/316840e12da9a815 FAIL (2025-12-27 07:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VVG4WFOKMXX2FMY276L3I4ZW3GYJJBNHVDXIFHHYL4RRE442V2PA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ZFKUB3ZM6J6EEF4NDOIKZFWZVIESE5SUJVWX4RNLOBQXLLLUQSQQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: XJP5TGA57E3SXAMP6UB577YJN76XN2WLVUOELICWUX5QXP2VJNZQ) context deadline exceeded ======LOGS====== 27 07:00:49.051038801 (519h31m48.922121846s) [upload=455797]: Starting upload Dec 27 07:00:49.051093896 (519h31m48.922176941s) [upload=455797]: Done starting upload Dec 27 07:00:49.051159561 (519h31m48.922242606s) [upload=455797]: Uploading segments... Dec 27 07:00:49.051273167 (519h31m48.922356203s) [upload=455797]: Got next segment. Inline: false Dec 27 07:00:49.051325958 (519h31m48.922408994s) [upload=455797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:00:49.051330557 (519h31m48.922413602s) [upload=455797]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:00:49.320639445 (519h31m49.19172248s) [seg_pos={0 0}, upload=455797]: Begin upload segment... Dec 27 07:00:49.320690141 (519h31m49.191773176s) [seg_pos={0 0}, upload=455797]: Done begin upload segment. Dec 27 07:00:49.320733494 (519h31m49.191816539s) [noise=true , address=45.91.116.6:4973 , piece_id=XJP5TGA57E3SXAMP6UB577YJN76XN2WLVUOELICWUX5QXP2VJNZQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.320757410 (519h31m49.191840455s) [noise=true , address=145.79.227.37:4565 , piece_id=24P75DDZUFUV7WK4B6Y42JO46MD5SEAAGQKQGDXG2QAYU4FSRRVA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.320801553 (519h31m49.191884598s) [noise=true , address=145.79.227.47:4265 , piece_id=AWO56KIT6PXOYL2YS3A4VDNWN4NNPDYB6A2TW4FU7CKH7ZLSZXZQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.321389032 (519h31m49.192472067s) [noise=true , address=77.90.31.17:4773 , piece_id=OCLGYW6UVKBIJJ2YYBQBUXOF3K3D44RRSKXXONRLDI37JN2QR6SQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.321782281 (519h31m49.192865326s) [noise=true , address=45.91.116.8:4048 , piece_id=HPMAPUDETNCNPEU3HHWW5SM4S43T7JNLTMXFVKY4KJGGPPBDMDKA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.322306610 (519h31m49.193389655s) [noise=true , address=145.79.227.11:4730 , piece_id=KWQSIAXDKDGDG2IUVZRIVXUQE6BIY2ELST6TDI7234PO5MSEUDAA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.322747489 (519h31m49.193830534s) [noise=true , address=145.79.227.15:4873 , piece_id=AUZGGAQLDFEDFOQ5UEUMNCHU6JIJ6IUNBITWTZQHIIKKQNHZ2EHQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.323325420 (519h31m49.194408455s) [noise=true , address=145.79.227.22:4510 , piece_id=DB3YZIBIB5E7DYRRGHHVQLPQ56AWUXGL343CJR5ZMHRWTQVRZUHQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.323876650 (519h31m49.194959695s) [noise=true , address=45.91.116.11:4118 , piece_id=RJRLHU642TIMM772PI43E32LYFZ46QVKHGFBGCAURCDYGJ5V35RQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.324536536 (519h31m49.195619581s) [noise=true , address=77.90.31.16:4736 , piece_id=VD7VF7SLPADS5DSV74B5H6YS352ZYCGVYHP4X47C6GTJG6ZWMCNQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.325787578 (519h31m49.196870623s) [noise=true , address=77.90.31.14:4664 , piece_id=VVG4WFOKMXX2FMY276L3I4ZW3GYJJBNHVDXIFHHYL4RRE442V2PA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.327132930 (519h31m49.198215966s) [noise=true , address=145.79.227.5:4221 , piece_id=74T4VJXCRIL4FZ4ALWTT4YHPMXXBWKU3IMYZ7IOF2ZGPIAKVE7TQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.329103753 (519h31m49.200186798s) [noise=true , address=145.79.227.6:4259 , piece_id=GB42TKIDAU6EWCANCTOAVZN7QI2CIGQEBCWQNNMJX3MHACRJNH2A, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.331084785 (519h31m49.20216782s) [noise=true , address=145.79.227.36:4518 , piece_id=R3KAQG6KRMBOBBA5HXNL57JY43QQC7UYGYNW66J7IZMQE7VYLS3A, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.332720039 (519h31m49.203803094s) [noise=true , address=77.90.31.11:4555 , piece_id=JPPQSNBMC336XVRHF3AIGDVI7SYDRFFNFJ5RWQZELJZ5Y4WXWT6A, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.337706164 (519h31m49.208789209s) [noise=true , address=45.91.116.9:4264 , piece_id=ZFKUB3ZM6J6EEF4NDOIKZFWZVIESE5SUJVWX4RNLOBQXLLLUQSQQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:51.049779524 (519h31m50.920862579s) [upload=455797]: Next returned nil segment Dec 27 07:00:51.049834658 (519h31m50.920917703s) [upload=455797]: Waiting for error group managing segments... Dec 27 07:00:51.336427749 (519h31m51.207510794s) [noise=true , address=145.79.227.15:4873 , piece_id=AUZGGAQLDFEDFOQ5UEUMNCHU6JIJ6IUNBITWTZQHIIKKQNHZ2EHQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.337125528 (519h31m51.208208573s) [noise=true , address=145.79.227.37:4565 , piece_id=24P75DDZUFUV7WK4B6Y42JO46MD5SEAAGQKQGDXG2QAYU4FSRRVA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.345051581 (519h31m51.216134626s) [noise=true , address=145.79.227.5:4221 , piece_id=74T4VJXCRIL4FZ4ALWTT4YHPMXXBWKU3IMYZ7IOF2ZGPIAKVE7TQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.348034280 (519h31m51.219117316s) [noise=true , address=145.79.227.47:4265 , piece_id=AWO56KIT6PXOYL2YS3A4VDNWN4NNPDYB6A2TW4FU7CKH7ZLSZXZQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.348764771 (519h31m51.219847847s) [noise=true , address=145.79.227.11:4730 , piece_id=KWQSIAXDKDGDG2IUVZRIVXUQE6BIY2ELST6TDI7234PO5MSEUDAA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.350353636 (519h31m51.221436671s) [noise=true , address=145.79.227.22:4510 , piece_id=DB3YZIBIB5E7DYRRGHHVQLPQ56AWUXGL343CJR5ZMHRWTQVRZUHQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.359355668 (519h31m51.230438713s) [noise=true , address=145.79.227.6:4259 , piece_id=GB42TKIDAU6EWCANCTOAVZN7QI2CIGQEBCWQNNMJX3MHACRJNH2A, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.365285059 (519h31m51.236368094s) [noise=true , address=145.79.227.36:4518 , piece_id=R3KAQG6KRMBOBBA5HXNL57JY43QQC7UYGYNW66J7IZMQE7VYLS3A, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.564273793 (519h31m51.435356828s) [noise=true , address=45.91.116.9:4264 , piece_id=ZFKUB3ZM6J6EEF4NDOIKZFWZVIESE5SUJVWX4RNLOBQXLLLUQSQQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.632836340 (519h31m51.503919385s) [noise=true , address=77.90.31.14:4664 , piece_id=VVG4WFOKMXX2FMY276L3I4ZW3GYJJBNHVDXIFHHYL4RRE442V2PA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.658615609 (519h31m51.529698654s) [noise=true , address=45.91.116.6:4973 , piece_id=XJP5TGA57E3SXAMP6UB577YJN76XN2WLVUOELICWUX5QXP2VJNZQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.737596855 (519h31m51.6086799s) [noise=true , address=45.91.116.11:4118 , piece_id=RJRLHU642TIMM772PI43E32LYFZ46QVKHGFBGCAURCDYGJ5V35RQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.960622530 (519h31m51.831705565s) [noise=true , address=77.90.31.11:4555 , piece_id=JPPQSNBMC336XVRHF3AIGDVI7SYDRFFNFJ5RWQZELJZ5Y4WXWT6A, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.960644462 (519h31m51.831727497s) [seg_pos={0 0}, upload=455797]: Segment reached optimal threshold of 13 pieces. Dec 27 07:00:51.960857908 (519h31m51.831940953s) [noise=true , address=77.90.31.16:4736 , piece_id=VD7VF7SLPADS5DSV74B5H6YS352ZYCGVYHP4X47C6GTJG6ZWMCNQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 07:00:51.960868107 (519h31m51.831951162s) [noise=true , address=77.90.31.16:4736 , piece_id=VD7VF7SLPADS5DSV74B5H6YS352ZYCGVYHP4X47C6GTJG6ZWMCNQ, seg_pos={0 0}, upload=455797]: Long tail cancellation due to optimal threshold Dec 27 07:00:51.960973899 (519h31m51.832056934s) [noise=true , address=77.90.31.17:4773 , piece_id=OCLGYW6UVKBIJJ2YYBQBUXOF3K3D44RRSKXXONRLDI37JN2QR6SQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:56524->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:56524->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56524->77.90.31.17:4773: use of closed network connection Dec 27 07:00:51.960990891 (519h31m51.832073936s) [noise=true , address=77.90.31.17:4773 , piece_id=OCLGYW6UVKBIJJ2YYBQBUXOF3K3D44RRSKXXONRLDI37JN2QR6SQ, seg_pos={0 0}, upload=455797]: Long tail cancellation due to optimal threshold Dec 27 07:00:51.961152900 (519h31m51.832235945s) [noise=true , address=45.91.116.8:4048 , piece_id=HPMAPUDETNCNPEU3HHWW5SM4S43T7JNLTMXFVKY4KJGGPPBDMDKA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 07:00:51.961162178 (519h31m51.832245213s) [noise=true , address=45.91.116.8:4048 , piece_id=HPMAPUDETNCNPEU3HHWW5SM4S43T7JNLTMXFVKY4KJGGPPBDMDKA, seg_pos={0 0}, upload=455797]: Long tail cancellation due to optimal threshold Dec 27 07:00:51.961180352 (519h31m51.832263397s) [seg_pos={0 0}, upload=455797]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:00:51.961249443 (519h31m51.832332488s) [upload=455797]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:00:51.961254163 (519h31m51.832337208s) [upload=455797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:00:52.259259713 (519h31m52.130342758s) [upload=455797]: Done uploading segments... Dec 27 07:00:52.259282738 (519h31m52.130365783s) [upload=455797]: Upload finished. err: <nil> Duration: 30.002394547s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...................................X......] duration: 54.406803529s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 07:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 54.406820661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X.....X.........X.....X.................X..] duration: 3.522128462s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:01:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522149322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X.....X.........X.....X.................X.] duration: 3.35444426s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:00:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.354469858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...................................X.....] duration: 50.892026398s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 07:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 50.892056295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X.....X.........X.....X.................X] duration: 5.001635405s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:00:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001663877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..] duration: 5.935029854s trace: https://jaeger.dev.storj.io/trace/7e7d6a774ae6269e FAIL (2025-12-27 06:59:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q) context deadline exceeded ======LOGS====== 8.265525271s) [upload=455759]: Starting upload Dec 27 06:57:58.394529532 (519h28m58.265612587s) [upload=455759]: Done starting upload Dec 27 06:57:58.394635985 (519h28m58.26571905s) [upload=455759]: Uploading segments... Dec 27 06:57:58.394880952 (519h28m58.265963987s) [upload=455759]: Got next segment. Inline: false Dec 27 06:57:58.394945263 (519h28m58.266028308s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:57:58.394952137 (519h28m58.266035192s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:57:58.639815164 (519h28m58.5108982s) [seg_pos={0 0}, upload=455759]: Begin upload segment... Dec 27 06:57:58.639867253 (519h28m58.510950298s) [seg_pos={0 0}, upload=455759]: Done begin upload segment. Dec 27 06:57:58.639893263 (519h28m58.510976298s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.639963336 (519h28m58.511046381s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.640729585 (519h28m58.51181262s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641104179 (519h28m58.512187214s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641480856 (519h28m58.512563891s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641980057 (519h28m58.513063092s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642061112 (519h28m58.513144147s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642478186 (519h28m58.513561231s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642545403 (519h28m58.513628438s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642863810 (519h28m58.513946855s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642994188 (519h28m58.514077233s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643145426 (519h28m58.514228461s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643540418 (519h28m58.514623473s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644160799 (519h28m58.515243835s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644560991 (519h28m58.515644026s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.646177860 (519h28m58.517260905s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:58:00.354941599 (519h29m0.226024634s) [upload=455759]: Next returned nil segment Dec 27 06:58:00.354954123 (519h29m0.226037168s) [upload=455759]: Waiting for error group managing segments... Dec 27 06:58:00.638651580 (519h29m0.509734615s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.644240012 (519h29m0.515323057s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.649912613 (519h29m0.520995658s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.650586967 (519h29m0.521670012s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.651635293 (519h29m0.522718328s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.654430717 (519h29m0.525513762s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.665700136 (519h29m0.536783171s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.668063316 (519h29m0.539146361s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.682023879 (519h29m0.553106914s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.834079984 (519h29m0.705163029s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.927458836 (519h29m0.798541881s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.197970073 (519h29m1.069053108s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572665028 (519h29m1.443748074s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572687861 (519h29m1.443770906s) [seg_pos={0 0}, upload=455759]: Segment reached optimal threshold of 13 pieces. Dec 27 06:58:01.572929111 (519h29m1.444012166s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:58:01.572938027 (519h29m1.444021073s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573021747 (519h29m1.444104793s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection Dec 27 06:58:01.573032648 (519h29m1.444115683s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573267155 (519h29m1.4443502s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:56680->77.90.31.16:4736: use of closed network connection; context canceled Dec 27 06:58:01.573277174 (519h29m1.444360229s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573313503 (519h29m1.444396548s) [seg_pos={0 0}, upload=455759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:58:01.573408814 (519h29m1.444491859s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:58:01.573415266 (519h29m1.444498312s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:58:01.866443211 (519h29m1.737526266s) [upload=455759]: Done uploading segments... Dec 27 06:58:01.866472807 (519h29m1.737555852s) [upload=455759]: Upload finished. err: <nil> Duration: 5.935065572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...................................X....] duration: 51.855900507s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 06:59:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 51.85591807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................X.....X.........X.....X.................] duration: 3.815964287s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:59:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815993664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X.] duration: 5.657723093s trace: https://jaeger.dev.storj.io/trace/7e7d6a774ae6269e FAIL (2025-12-27 06:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q) context deadline exceeded ======LOGS====== 8.265525271s) [upload=455759]: Starting upload Dec 27 06:57:58.394529532 (519h28m58.265612587s) [upload=455759]: Done starting upload Dec 27 06:57:58.394635985 (519h28m58.26571905s) [upload=455759]: Uploading segments... Dec 27 06:57:58.394880952 (519h28m58.265963987s) [upload=455759]: Got next segment. Inline: false Dec 27 06:57:58.394945263 (519h28m58.266028308s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:57:58.394952137 (519h28m58.266035192s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:57:58.639815164 (519h28m58.5108982s) [seg_pos={0 0}, upload=455759]: Begin upload segment... Dec 27 06:57:58.639867253 (519h28m58.510950298s) [seg_pos={0 0}, upload=455759]: Done begin upload segment. Dec 27 06:57:58.639893263 (519h28m58.510976298s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.639963336 (519h28m58.511046381s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.640729585 (519h28m58.51181262s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641104179 (519h28m58.512187214s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641480856 (519h28m58.512563891s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641980057 (519h28m58.513063092s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642061112 (519h28m58.513144147s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642478186 (519h28m58.513561231s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642545403 (519h28m58.513628438s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642863810 (519h28m58.513946855s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642994188 (519h28m58.514077233s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643145426 (519h28m58.514228461s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643540418 (519h28m58.514623473s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644160799 (519h28m58.515243835s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644560991 (519h28m58.515644026s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.646177860 (519h28m58.517260905s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:58:00.354941599 (519h29m0.226024634s) [upload=455759]: Next returned nil segment Dec 27 06:58:00.354954123 (519h29m0.226037168s) [upload=455759]: Waiting for error group managing segments... Dec 27 06:58:00.638651580 (519h29m0.509734615s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.644240012 (519h29m0.515323057s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.649912613 (519h29m0.520995658s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.650586967 (519h29m0.521670012s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.651635293 (519h29m0.522718328s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.654430717 (519h29m0.525513762s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.665700136 (519h29m0.536783171s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.668063316 (519h29m0.539146361s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.682023879 (519h29m0.553106914s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.834079984 (519h29m0.705163029s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.927458836 (519h29m0.798541881s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.197970073 (519h29m1.069053108s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572665028 (519h29m1.443748074s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572687861 (519h29m1.443770906s) [seg_pos={0 0}, upload=455759]: Segment reached optimal threshold of 13 pieces. Dec 27 06:58:01.572929111 (519h29m1.444012166s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:58:01.572938027 (519h29m1.444021073s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573021747 (519h29m1.444104793s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection Dec 27 06:58:01.573032648 (519h29m1.444115683s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573267155 (519h29m1.4443502s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:56680->77.90.31.16:4736: use of closed network connection; context canceled Dec 27 06:58:01.573277174 (519h29m1.444360229s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573313503 (519h29m1.444396548s) [seg_pos={0 0}, upload=455759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:58:01.573408814 (519h29m1.444491859s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:58:01.573415266 (519h29m1.444498312s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:58:01.866443211 (519h29m1.737526266s) [upload=455759]: Done uploading segments... Dec 27 06:58:01.866472807 (519h29m1.737555852s) [upload=455759]: Upload finished. err: <nil> Duration: 5.657749964s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X.....X.........X.....X................] duration: 3.604090991s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:58:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.604121229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X] duration: 30.002885479s trace: https://jaeger.dev.storj.io/trace/7e7d6a774ae6269e FAIL (2025-12-27 06:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q) context deadline exceeded ======LOGS====== 8.265525271s) [upload=455759]: Starting upload Dec 27 06:57:58.394529532 (519h28m58.265612587s) [upload=455759]: Done starting upload Dec 27 06:57:58.394635985 (519h28m58.26571905s) [upload=455759]: Uploading segments... Dec 27 06:57:58.394880952 (519h28m58.265963987s) [upload=455759]: Got next segment. Inline: false Dec 27 06:57:58.394945263 (519h28m58.266028308s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:57:58.394952137 (519h28m58.266035192s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:57:58.639815164 (519h28m58.5108982s) [seg_pos={0 0}, upload=455759]: Begin upload segment... Dec 27 06:57:58.639867253 (519h28m58.510950298s) [seg_pos={0 0}, upload=455759]: Done begin upload segment. Dec 27 06:57:58.639893263 (519h28m58.510976298s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.639963336 (519h28m58.511046381s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.640729585 (519h28m58.51181262s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641104179 (519h28m58.512187214s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641480856 (519h28m58.512563891s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641980057 (519h28m58.513063092s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642061112 (519h28m58.513144147s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642478186 (519h28m58.513561231s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642545403 (519h28m58.513628438s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642863810 (519h28m58.513946855s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642994188 (519h28m58.514077233s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643145426 (519h28m58.514228461s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643540418 (519h28m58.514623473s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644160799 (519h28m58.515243835s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644560991 (519h28m58.515644026s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.646177860 (519h28m58.517260905s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:58:00.354941599 (519h29m0.226024634s) [upload=455759]: Next returned nil segment Dec 27 06:58:00.354954123 (519h29m0.226037168s) [upload=455759]: Waiting for error group managing segments... Dec 27 06:58:00.638651580 (519h29m0.509734615s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.644240012 (519h29m0.515323057s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.649912613 (519h29m0.520995658s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.650586967 (519h29m0.521670012s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.651635293 (519h29m0.522718328s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.654430717 (519h29m0.525513762s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.665700136 (519h29m0.536783171s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.668063316 (519h29m0.539146361s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.682023879 (519h29m0.553106914s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.834079984 (519h29m0.705163029s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.927458836 (519h29m0.798541881s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.197970073 (519h29m1.069053108s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572665028 (519h29m1.443748074s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572687861 (519h29m1.443770906s) [seg_pos={0 0}, upload=455759]: Segment reached optimal threshold of 13 pieces. Dec 27 06:58:01.572929111 (519h29m1.444012166s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:58:01.572938027 (519h29m1.444021073s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573021747 (519h29m1.444104793s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection Dec 27 06:58:01.573032648 (519h29m1.444115683s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573267155 (519h29m1.4443502s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:56680->77.90.31.16:4736: use of closed network connection; context canceled Dec 27 06:58:01.573277174 (519h29m1.444360229s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573313503 (519h29m1.444396548s) [seg_pos={0 0}, upload=455759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:58:01.573408814 (519h29m1.444491859s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:58:01.573415266 (519h29m1.444498312s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:58:01.866443211 (519h29m1.737526266s) [upload=455759]: Done uploading segments... Dec 27 06:58:01.866472807 (519h29m1.737555852s) [upload=455759]: Upload finished. err: <nil> Duration: 30.002910516s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X.....X.........X.....X...............] duration: 5.001104178s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:58:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001123605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...................................X...] duration: 43.913373835s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 06:58:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 43.91338756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...................................X..] duration: 55.087278929s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 06:57:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 55.08730062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X.....X.........X.....X..............] duration: 3.489716308s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:57:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489740423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..] duration: 6.007333715s trace: https://jaeger.dev.storj.io/trace/24db2465011bc0da FAIL (2025-12-27 06:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA) context deadline exceeded ======LOGS====== Dec 27 06:55:32.801480142 (519h26m32.672563187s) [upload=455724]: Starting upload Dec 27 06:55:32.801546247 (519h26m32.672629292s) [upload=455724]: Done starting upload Dec 27 06:55:32.801610981 (519h26m32.672694026s) [upload=455724]: Uploading segments... Dec 27 06:55:32.801735408 (519h26m32.672818443s) [upload=455724]: Got next segment. Inline: false Dec 27 06:55:32.801782257 (519h26m32.672865292s) [upload=455724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:55:32.801796214 (519h26m32.672879259s) [upload=455724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:55:33.046824838 (519h26m32.917907873s) [seg_pos={0 0}, upload=455724]: Begin upload segment... Dec 27 06:55:33.046875455 (519h26m32.9179585s) [seg_pos={0 0}, upload=455724]: Done begin upload segment. Dec 27 06:55:33.046912585 (519h26m32.91799563s) [noise=true , address=145.79.227.17:4971 , piece_id=XXFU45PMMH3OBUQVDIKUQZHZ7IGCY7KMW5GDGQDOJHWDJ6KWAVXQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.047051941 (519h26m32.918134986s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.047739120 (519h26m32.918822165s) [noise=true , address=145.79.227.11:4729 , piece_id=PGAYGJRHGNKQYVTNSS7H7434VEXQNWDXX752Q2LNNLX5AF7VHXCQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.048168176 (519h26m32.919251211s) [noise=true , address=145.79.227.2:4113 , piece_id=DFLI5MRZKKVOAHMQ57Z7JMTARVLCNALIFOILCKYD7BVRCUOFU2LA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.048593326 (519h26m32.919676371s) [noise=true , address=145.79.227.48:4302 , piece_id=AKUZ5Q3WIUBBZPGFPB7HAQEBO6VJH2S2MEDGJ7P5Q5HMILVVHVPQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.049025118 (519h26m32.920108163s) [noise=true , address=145.79.227.42:4122 , piece_id=45QYGDKTEO2AC6IJPZZJOBYLIUDDRRATJGP67L45Z5BFSVPNBBFQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.049447142 (519h26m32.920530187s) [noise=true , address=145.79.227.5:4221 , piece_id=CBGVAKH4ZQFYSXGQWGTXELQDCJYPLTYWBOE6VGUNEZZQHSSXUXJA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.051238292 (519h26m32.922321337s) [noise=true , address=45.91.116.6:4975 , piece_id=ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.051312643 (519h26m32.922395689s) [noise=true , address=145.79.227.12:4765 , piece_id=WWDJLBR7GGGEG2RJESKXSUJROS2GZHKRUDCMN4NDHG6VN4KMXVHQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.054001614 (519h26m32.92508466s) [noise=true , address=145.79.227.52:4445 , piece_id=3CVL6BGNG2UNZHVOLK7ACW7MZN5NPYTI2MUFIPCN3OJUKGDMUTDQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.055559741 (519h26m32.926642776s) [noise=true , address=145.79.227.24:4581 , piece_id=J5HCEIZSAUDM45CFB3EMPN6H43UAHDFUGK5ZYDSY5EDRLOVT4M4A, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.056473712 (519h26m32.927556757s) [noise=true , address=145.79.227.47:4265 , piece_id=UHXHJRDTX6ACVNJHXQDELIYCBMSWJ4QPIJWC225P6ADO5CZR2XMQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.058173948 (519h26m32.929256984s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.059478012 (519h26m32.930561047s) [noise=true , address=45.91.116.13:4156 , piece_id=2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.061666089 (519h26m32.932749134s) [noise=true , address=77.90.31.13:4627 , piece_id=HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.062167663 (519h26m32.933250709s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:34.769439009 (519h26m34.640522054s) [upload=455724]: Next returned nil segment Dec 27 06:55:34.769454590 (519h26m34.640537625s) [upload=455724]: Waiting for error group managing segments... Dec 27 06:55:35.059794144 (519h26m34.930877189s) [noise=true , address=145.79.227.42:4122 , piece_id=45QYGDKTEO2AC6IJPZZJOBYLIUDDRRATJGP67L45Z5BFSVPNBBFQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.063596565 (519h26m34.9346796s) [noise=true , address=145.79.227.48:4302 , piece_id=AKUZ5Q3WIUBBZPGFPB7HAQEBO6VJH2S2MEDGJ7P5Q5HMILVVHVPQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.064391117 (519h26m34.935474162s) [noise=true , address=145.79.227.2:4113 , piece_id=DFLI5MRZKKVOAHMQ57Z7JMTARVLCNALIFOILCKYD7BVRCUOFU2LA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.065920239 (519h26m34.937003274s) [noise=true , address=145.79.227.17:4971 , piece_id=XXFU45PMMH3OBUQVDIKUQZHZ7IGCY7KMW5GDGQDOJHWDJ6KWAVXQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.067860223 (519h26m34.938943258s) [noise=true , address=145.79.227.12:4765 , piece_id=WWDJLBR7GGGEG2RJESKXSUJROS2GZHKRUDCMN4NDHG6VN4KMXVHQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.068559284 (519h26m34.939642319s) [noise=true , address=145.79.227.52:4445 , piece_id=3CVL6BGNG2UNZHVOLK7ACW7MZN5NPYTI2MUFIPCN3OJUKGDMUTDQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.069202970 (519h26m34.940286005s) [noise=true , address=145.79.227.5:4221 , piece_id=CBGVAKH4ZQFYSXGQWGTXELQDCJYPLTYWBOE6VGUNEZZQHSSXUXJA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.069245531 (519h26m34.940328566s) [noise=true , address=145.79.227.11:4729 , piece_id=PGAYGJRHGNKQYVTNSS7H7434VEXQNWDXX752Q2LNNLX5AF7VHXCQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.072294747 (519h26m34.943377792s) [noise=true , address=145.79.227.24:4581 , piece_id=J5HCEIZSAUDM45CFB3EMPN6H43UAHDFUGK5ZYDSY5EDRLOVT4M4A, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.104530491 (519h26m34.975613526s) [noise=true , address=145.79.227.47:4265 , piece_id=UHXHJRDTX6ACVNJHXQDELIYCBMSWJ4QPIJWC225P6ADO5CZR2XMQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.258133421 (519h26m35.129216456s) [noise=true , address=77.90.31.13:4627 , piece_id=HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.341496000 (519h26m35.212579035s) [noise=true , address=45.91.116.6:4975 , piece_id=ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.350723101 (519h26m35.221806146s) [noise=true , address=45.91.116.13:4156 , piece_id=2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.350749250 (519h26m35.221832285s) [seg_pos={0 0}, upload=455724]: Segment reached optimal threshold of 13 pieces. Dec 27 06:55:35.350997934 (519h26m35.222080989s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 06:55:35.351009556 (519h26m35.222092601s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351128072 (519h26m35.222211117s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection Dec 27 06:55:35.351138823 (519h26m35.222221858s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351390822 (519h26m35.222473858s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 27 06:55:35.351399819 (519h26m35.222482864s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351420459 (519h26m35.222503504s) [seg_pos={0 0}, upload=455724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:55:35.351537231 (519h26m35.222620286s) [upload=455724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:55:35.351545707 (519h26m35.222628752s) [upload=455724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:55:35.637477909 (519h26m35.508560944s) [upload=455724]: Done uploading segments... Dec 27 06:55:35.637496695 (519h26m35.50857973s) [upload=455724]: Upload finished. err: <nil> Duration: 6.007361559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X.....X.........X.....X.............] duration: 3.675043067s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:56:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.675068735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX.] duration: 6.009281255s trace: https://jaeger.dev.storj.io/trace/24db2465011bc0da FAIL (2025-12-27 06:56:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA) context deadline exceeded ======LOGS====== Dec 27 06:55:32.801480142 (519h26m32.672563187s) [upload=455724]: Starting upload Dec 27 06:55:32.801546247 (519h26m32.672629292s) [upload=455724]: Done starting upload Dec 27 06:55:32.801610981 (519h26m32.672694026s) [upload=455724]: Uploading segments... Dec 27 06:55:32.801735408 (519h26m32.672818443s) [upload=455724]: Got next segment. Inline: false Dec 27 06:55:32.801782257 (519h26m32.672865292s) [upload=455724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:55:32.801796214 (519h26m32.672879259s) [upload=455724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:55:33.046824838 (519h26m32.917907873s) [seg_pos={0 0}, upload=455724]: Begin upload segment... Dec 27 06:55:33.046875455 (519h26m32.9179585s) [seg_pos={0 0}, upload=455724]: Done begin upload segment. Dec 27 06:55:33.046912585 (519h26m32.91799563s) [noise=true , address=145.79.227.17:4971 , piece_id=XXFU45PMMH3OBUQVDIKUQZHZ7IGCY7KMW5GDGQDOJHWDJ6KWAVXQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.047051941 (519h26m32.918134986s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.047739120 (519h26m32.918822165s) [noise=true , address=145.79.227.11:4729 , piece_id=PGAYGJRHGNKQYVTNSS7H7434VEXQNWDXX752Q2LNNLX5AF7VHXCQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.048168176 (519h26m32.919251211s) [noise=true , address=145.79.227.2:4113 , piece_id=DFLI5MRZKKVOAHMQ57Z7JMTARVLCNALIFOILCKYD7BVRCUOFU2LA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.048593326 (519h26m32.919676371s) [noise=true , address=145.79.227.48:4302 , piece_id=AKUZ5Q3WIUBBZPGFPB7HAQEBO6VJH2S2MEDGJ7P5Q5HMILVVHVPQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.049025118 (519h26m32.920108163s) [noise=true , address=145.79.227.42:4122 , piece_id=45QYGDKTEO2AC6IJPZZJOBYLIUDDRRATJGP67L45Z5BFSVPNBBFQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.049447142 (519h26m32.920530187s) [noise=true , address=145.79.227.5:4221 , piece_id=CBGVAKH4ZQFYSXGQWGTXELQDCJYPLTYWBOE6VGUNEZZQHSSXUXJA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.051238292 (519h26m32.922321337s) [noise=true , address=45.91.116.6:4975 , piece_id=ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.051312643 (519h26m32.922395689s) [noise=true , address=145.79.227.12:4765 , piece_id=WWDJLBR7GGGEG2RJESKXSUJROS2GZHKRUDCMN4NDHG6VN4KMXVHQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.054001614 (519h26m32.92508466s) [noise=true , address=145.79.227.52:4445 , piece_id=3CVL6BGNG2UNZHVOLK7ACW7MZN5NPYTI2MUFIPCN3OJUKGDMUTDQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.055559741 (519h26m32.926642776s) [noise=true , address=145.79.227.24:4581 , piece_id=J5HCEIZSAUDM45CFB3EMPN6H43UAHDFUGK5ZYDSY5EDRLOVT4M4A, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.056473712 (519h26m32.927556757s) [noise=true , address=145.79.227.47:4265 , piece_id=UHXHJRDTX6ACVNJHXQDELIYCBMSWJ4QPIJWC225P6ADO5CZR2XMQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.058173948 (519h26m32.929256984s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.059478012 (519h26m32.930561047s) [noise=true , address=45.91.116.13:4156 , piece_id=2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.061666089 (519h26m32.932749134s) [noise=true , address=77.90.31.13:4627 , piece_id=HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.062167663 (519h26m32.933250709s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:34.769439009 (519h26m34.640522054s) [upload=455724]: Next returned nil segment Dec 27 06:55:34.769454590 (519h26m34.640537625s) [upload=455724]: Waiting for error group managing segments... Dec 27 06:55:35.059794144 (519h26m34.930877189s) [noise=true , address=145.79.227.42:4122 , piece_id=45QYGDKTEO2AC6IJPZZJOBYLIUDDRRATJGP67L45Z5BFSVPNBBFQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.063596565 (519h26m34.9346796s) [noise=true , address=145.79.227.48:4302 , piece_id=AKUZ5Q3WIUBBZPGFPB7HAQEBO6VJH2S2MEDGJ7P5Q5HMILVVHVPQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.064391117 (519h26m34.935474162s) [noise=true , address=145.79.227.2:4113 , piece_id=DFLI5MRZKKVOAHMQ57Z7JMTARVLCNALIFOILCKYD7BVRCUOFU2LA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.065920239 (519h26m34.937003274s) [noise=true , address=145.79.227.17:4971 , piece_id=XXFU45PMMH3OBUQVDIKUQZHZ7IGCY7KMW5GDGQDOJHWDJ6KWAVXQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.067860223 (519h26m34.938943258s) [noise=true , address=145.79.227.12:4765 , piece_id=WWDJLBR7GGGEG2RJESKXSUJROS2GZHKRUDCMN4NDHG6VN4KMXVHQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.068559284 (519h26m34.939642319s) [noise=true , address=145.79.227.52:4445 , piece_id=3CVL6BGNG2UNZHVOLK7ACW7MZN5NPYTI2MUFIPCN3OJUKGDMUTDQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.069202970 (519h26m34.940286005s) [noise=true , address=145.79.227.5:4221 , piece_id=CBGVAKH4ZQFYSXGQWGTXELQDCJYPLTYWBOE6VGUNEZZQHSSXUXJA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.069245531 (519h26m34.940328566s) [noise=true , address=145.79.227.11:4729 , piece_id=PGAYGJRHGNKQYVTNSS7H7434VEXQNWDXX752Q2LNNLX5AF7VHXCQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.072294747 (519h26m34.943377792s) [noise=true , address=145.79.227.24:4581 , piece_id=J5HCEIZSAUDM45CFB3EMPN6H43UAHDFUGK5ZYDSY5EDRLOVT4M4A, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.104530491 (519h26m34.975613526s) [noise=true , address=145.79.227.47:4265 , piece_id=UHXHJRDTX6ACVNJHXQDELIYCBMSWJ4QPIJWC225P6ADO5CZR2XMQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.258133421 (519h26m35.129216456s) [noise=true , address=77.90.31.13:4627 , piece_id=HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.341496000 (519h26m35.212579035s) [noise=true , address=45.91.116.6:4975 , piece_id=ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.350723101 (519h26m35.221806146s) [noise=true , address=45.91.116.13:4156 , piece_id=2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.350749250 (519h26m35.221832285s) [seg_pos={0 0}, upload=455724]: Segment reached optimal threshold of 13 pieces. Dec 27 06:55:35.350997934 (519h26m35.222080989s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 06:55:35.351009556 (519h26m35.222092601s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351128072 (519h26m35.222211117s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection Dec 27 06:55:35.351138823 (519h26m35.222221858s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351390822 (519h26m35.222473858s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 27 06:55:35.351399819 (519h26m35.222482864s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351420459 (519h26m35.222503504s) [seg_pos={0 0}, upload=455724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:55:35.351537231 (519h26m35.222620286s) [upload=455724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:55:35.351545707 (519h26m35.222628752s) [upload=455724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:55:35.637477909 (519h26m35.508560944s) [upload=455724]: Done uploading segments... Dec 27 06:55:35.637496695 (519h26m35.50857973s) [upload=455724]: Upload finished. err: <nil> Duration: 6.009310701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...................................X.] duration: 44.058335652s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 06:56:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 44.058354768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX] duration: 30.003437684s trace: https://jaeger.dev.storj.io/trace/24db2465011bc0da FAIL (2025-12-27 06:56:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA) context deadline exceeded ======LOGS====== Dec 27 06:55:32.801480142 (519h26m32.672563187s) [upload=455724]: Starting upload Dec 27 06:55:32.801546247 (519h26m32.672629292s) [upload=455724]: Done starting upload Dec 27 06:55:32.801610981 (519h26m32.672694026s) [upload=455724]: Uploading segments... Dec 27 06:55:32.801735408 (519h26m32.672818443s) [upload=455724]: Got next segment. Inline: false Dec 27 06:55:32.801782257 (519h26m32.672865292s) [upload=455724]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:55:32.801796214 (519h26m32.672879259s) [upload=455724]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:55:33.046824838 (519h26m32.917907873s) [seg_pos={0 0}, upload=455724]: Begin upload segment... Dec 27 06:55:33.046875455 (519h26m32.9179585s) [seg_pos={0 0}, upload=455724]: Done begin upload segment. Dec 27 06:55:33.046912585 (519h26m32.91799563s) [noise=true , address=145.79.227.17:4971 , piece_id=XXFU45PMMH3OBUQVDIKUQZHZ7IGCY7KMW5GDGQDOJHWDJ6KWAVXQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.047051941 (519h26m32.918134986s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.047739120 (519h26m32.918822165s) [noise=true , address=145.79.227.11:4729 , piece_id=PGAYGJRHGNKQYVTNSS7H7434VEXQNWDXX752Q2LNNLX5AF7VHXCQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.048168176 (519h26m32.919251211s) [noise=true , address=145.79.227.2:4113 , piece_id=DFLI5MRZKKVOAHMQ57Z7JMTARVLCNALIFOILCKYD7BVRCUOFU2LA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.048593326 (519h26m32.919676371s) [noise=true , address=145.79.227.48:4302 , piece_id=AKUZ5Q3WIUBBZPGFPB7HAQEBO6VJH2S2MEDGJ7P5Q5HMILVVHVPQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.049025118 (519h26m32.920108163s) [noise=true , address=145.79.227.42:4122 , piece_id=45QYGDKTEO2AC6IJPZZJOBYLIUDDRRATJGP67L45Z5BFSVPNBBFQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.049447142 (519h26m32.920530187s) [noise=true , address=145.79.227.5:4221 , piece_id=CBGVAKH4ZQFYSXGQWGTXELQDCJYPLTYWBOE6VGUNEZZQHSSXUXJA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.051238292 (519h26m32.922321337s) [noise=true , address=45.91.116.6:4975 , piece_id=ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.051312643 (519h26m32.922395689s) [noise=true , address=145.79.227.12:4765 , piece_id=WWDJLBR7GGGEG2RJESKXSUJROS2GZHKRUDCMN4NDHG6VN4KMXVHQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.054001614 (519h26m32.92508466s) [noise=true , address=145.79.227.52:4445 , piece_id=3CVL6BGNG2UNZHVOLK7ACW7MZN5NPYTI2MUFIPCN3OJUKGDMUTDQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.055559741 (519h26m32.926642776s) [noise=true , address=145.79.227.24:4581 , piece_id=J5HCEIZSAUDM45CFB3EMPN6H43UAHDFUGK5ZYDSY5EDRLOVT4M4A, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.056473712 (519h26m32.927556757s) [noise=true , address=145.79.227.47:4265 , piece_id=UHXHJRDTX6ACVNJHXQDELIYCBMSWJ4QPIJWC225P6ADO5CZR2XMQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.058173948 (519h26m32.929256984s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.059478012 (519h26m32.930561047s) [noise=true , address=45.91.116.13:4156 , piece_id=2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.061666089 (519h26m32.932749134s) [noise=true , address=77.90.31.13:4627 , piece_id=HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:33.062167663 (519h26m32.933250709s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Uploading piece... Dec 27 06:55:34.769439009 (519h26m34.640522054s) [upload=455724]: Next returned nil segment Dec 27 06:55:34.769454590 (519h26m34.640537625s) [upload=455724]: Waiting for error group managing segments... Dec 27 06:55:35.059794144 (519h26m34.930877189s) [noise=true , address=145.79.227.42:4122 , piece_id=45QYGDKTEO2AC6IJPZZJOBYLIUDDRRATJGP67L45Z5BFSVPNBBFQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.063596565 (519h26m34.9346796s) [noise=true , address=145.79.227.48:4302 , piece_id=AKUZ5Q3WIUBBZPGFPB7HAQEBO6VJH2S2MEDGJ7P5Q5HMILVVHVPQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.064391117 (519h26m34.935474162s) [noise=true , address=145.79.227.2:4113 , piece_id=DFLI5MRZKKVOAHMQ57Z7JMTARVLCNALIFOILCKYD7BVRCUOFU2LA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.065920239 (519h26m34.937003274s) [noise=true , address=145.79.227.17:4971 , piece_id=XXFU45PMMH3OBUQVDIKUQZHZ7IGCY7KMW5GDGQDOJHWDJ6KWAVXQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.067860223 (519h26m34.938943258s) [noise=true , address=145.79.227.12:4765 , piece_id=WWDJLBR7GGGEG2RJESKXSUJROS2GZHKRUDCMN4NDHG6VN4KMXVHQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.068559284 (519h26m34.939642319s) [noise=true , address=145.79.227.52:4445 , piece_id=3CVL6BGNG2UNZHVOLK7ACW7MZN5NPYTI2MUFIPCN3OJUKGDMUTDQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.069202970 (519h26m34.940286005s) [noise=true , address=145.79.227.5:4221 , piece_id=CBGVAKH4ZQFYSXGQWGTXELQDCJYPLTYWBOE6VGUNEZZQHSSXUXJA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.069245531 (519h26m34.940328566s) [noise=true , address=145.79.227.11:4729 , piece_id=PGAYGJRHGNKQYVTNSS7H7434VEXQNWDXX752Q2LNNLX5AF7VHXCQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.072294747 (519h26m34.943377792s) [noise=true , address=145.79.227.24:4581 , piece_id=J5HCEIZSAUDM45CFB3EMPN6H43UAHDFUGK5ZYDSY5EDRLOVT4M4A, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.104530491 (519h26m34.975613526s) [noise=true , address=145.79.227.47:4265 , piece_id=UHXHJRDTX6ACVNJHXQDELIYCBMSWJ4QPIJWC225P6ADO5CZR2XMQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.258133421 (519h26m35.129216456s) [noise=true , address=77.90.31.13:4627 , piece_id=HFFBAQJ4AR5SY7UWNLMCHP243CDAXCXFQCYDMX6NGVNDKIVWZX6A, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.341496000 (519h26m35.212579035s) [noise=true , address=45.91.116.6:4975 , piece_id=ZYYH4KI45P5NMITHZQIBURRFUOJS6QHXGY4CUMNLXSX2SHLQHRRA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.350723101 (519h26m35.221806146s) [noise=true , address=45.91.116.13:4156 , piece_id=2C6V5VSR77D452MJZCHNDBKPP6AWLX4CBJQ7B4TJXMWUAG6KZKGQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: <nil> Dec 27 06:55:35.350749250 (519h26m35.221832285s) [seg_pos={0 0}, upload=455724]: Segment reached optimal threshold of 13 pieces. Dec 27 06:55:35.350997934 (519h26m35.222080989s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 06:55:35.351009556 (519h26m35.222092601s) [noise=true , address=45.91.116.4:4937 , piece_id=5AND32TKGLO5H5TPN6YZ7MN52WGUTV3UH6OTICKP2AMGX32QLSCA, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351128072 (519h26m35.222211117s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48672->77.90.31.11:4553: use of closed network connection Dec 27 06:55:35.351138823 (519h26m35.222221858s) [noise=true , address=77.90.31.11:4553 , piece_id=3QDRARNGG43P3BILXRCFYGNHPEZ3HFIFFUWXCCPGBHSGDNZY2BZA, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351390822 (519h26m35.222473858s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 27 06:55:35.351399819 (519h26m35.222482864s) [noise=true , address=145.79.227.16:4911 , piece_id=C3AULZ7HVJ7OUPWYRJZB7PGRIGNRSVEEFCOKOY767MGF2UMRWDZQ, seg_pos={0 0}, upload=455724]: Long tail cancellation due to optimal threshold Dec 27 06:55:35.351420459 (519h26m35.222503504s) [seg_pos={0 0}, upload=455724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:55:35.351537231 (519h26m35.222620286s) [upload=455724]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:55:35.351545707 (519h26m35.222628752s) [upload=455724]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:55:35.637477909 (519h26m35.508560944s) [upload=455724]: Done uploading segments... Dec 27 06:55:35.637496695 (519h26m35.50857973s) [upload=455724]: Upload finished. err: <nil> Duration: 30.003463714s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................X.....X.........X.....X............] duration: 3.629548216s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:56:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629901289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...................................X] duration: 1m0.003129303s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 06:55:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 1m0.003142629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................X.....X.........X.....X...........] duration: 3.484108759s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:55:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48413527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.X] duration: 30.003014651s trace: https://jaeger.dev.storj.io/trace/b78799f479bca5e FAIL (2025-12-27 06:54:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: UK73OUYF2D2QQU2J5TBYGB45PYZBDY7BE5I6XTQ3VAUQSSHACW2A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: JZAGUVLG6NJWOQK5ON7Z7FGOR7GYBJBOI6GKNR6CHIEJKYQZSNTQ) context deadline exceeded ======LOGS====== Dec 27 06:54:25.488381265 (519h25m25.359464311s) [upload=455713]: Starting upload Dec 27 06:54:25.488435899 (519h25m25.359518954s) [upload=455713]: Done starting upload Dec 27 06:54:25.488562220 (519h25m25.359645285s) [upload=455713]: Uploading segments... Dec 27 06:54:25.488691506 (519h25m25.359774551s) [upload=455713]: Got next segment. Inline: false Dec 27 06:54:25.488730901 (519h25m25.359813936s) [upload=455713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:54:25.488735159 (519h25m25.359818194s) [upload=455713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:54:25.732509405 (519h25m25.60359244s) [seg_pos={0 0}, upload=455713]: Begin upload segment... Dec 27 06:54:25.732561804 (519h25m25.603644839s) [seg_pos={0 0}, upload=455713]: Done begin upload segment. Dec 27 06:54:25.732602632 (519h25m25.603685667s) [noise=true , address=145.79.227.19:4079 , piece_id=3MAJKX64LBC32QMBNOLP5CM4GLORBTANGJQ5ME7A4EC6WLH74VMA, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.732676814 (519h25m25.603759859s) [noise=true , address=145.79.227.15:4873 , piece_id=MUS34ISVIR67SYD6GT2V7MLEDJHRKH3CQQECXMFVMJC6YO7RBGYQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.733286584 (519h25m25.604369629s) [noise=true , address=77.90.31.18:4808 , piece_id=5MGRYJQ2GC2VZJFPY4P5OODYX42BLJYPS7GZ7BFA7FWUU3SDJBUA, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.733704700 (519h25m25.604787745s) [noise=true , address=145.79.227.3:4151 , piece_id=M263OMEQHOENHQ36XCPLPKG6OBQMBYIQRV5VAZC5X7JYB3I6O5QQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.734130942 (519h25m25.605213977s) [noise=true , address=145.79.227.12:4765 , piece_id=SXDZSFYNG3GQS2UJX47CYL7RVCJUVD2K6H6LYZNLGIZBSZKLVLQQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.734514863 (519h25m25.605597908s) [noise=true , address=145.79.227.47:4266 , piece_id=SQKABZSWUXJXXIYBXMDPXX3VNIHHYMESJHXHRNRR4TOQPR735OFQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.734978917 (519h25m25.606061952s) [noise=true , address=145.79.227.36:4519 , piece_id=FYEMUY3GKOKZ6QQEQ23SE5WHJLFNUNEERYDJQHJBRW76ROLBA7WQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.734988575 (519h25m25.60607161s) [noise=true , address=45.91.116.7:4010 , piece_id=UK73OUYF2D2QQU2J5TBYGB45PYZBDY7BE5I6XTQ3VAUQSSHACW2A, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.735313764 (519h25m25.606396799s) [noise=true , address=77.90.31.21:4878 , piece_id=JZAGUVLG6NJWOQK5ON7Z7FGOR7GYBJBOI6GKNR6CHIEJKYQZSNTQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.735445525 (519h25m25.60652857s) [noise=true , address=45.91.116.11:4120 , piece_id=FEOQVQXQQ7AZ63HKULHWYLOYATJRXWM4CK4DJ3QBOFXZIWAWPN3A, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.735697635 (519h25m25.60678067s) [noise=true , address=145.79.227.42:4121 , piece_id=DNPVOU6BMRQ24PYZQCT6OSWXXW2X7SMGGA5OGLPL3AVCM7W733FQ, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.735964223 (519h25m25.607047268s) [noise=true , address=145.79.227.50:4373 , piece_id=PG7D6WIGWLEUP357R2UNBGQHZ53LH3PE4XJRVCPMSSSIJJAX455A, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.736023696 (519h25m25.607106741s) [noise=true , address=145.79.227.4:4186 , piece_id=FORBLNPBGKCCQIJIF4UJ2N7IMTV6KRLN4MTNNEIM6Q5A7Z34WM6A, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.736347052 (519h25m25.607430087s) [noise=true , address=145.79.227.48:4301 , piece_id=KPJMMTOAGUNQKNQSO5FRHRAARUUCZ4JBBP3EFE2LW6PUOCOUYTGA, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.736660479 (519h25m25.607743524s) [noise=true , address=145.79.227.38:4977 , piece_id=K7TTGWKUEOT2AC5QKYNBOMISX2ZLWKP53LYLPH6GIYXTNYHOZ3YA, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:25.737006417 (519h25m25.608089453s) [noise=true , address=145.79.227.34:4905 , piece_id=6LX6ZHFFKNUMW5CLJIWLMPGLGMN6NVIZ5MEXQWULMMGHQHN3AC7A, seg_pos={0 0}, upload=455713]: Uploading piece... Dec 27 06:54:27.443816295 (519h25m27.31489933s) [upload=455713]: Next returned nil segment Dec 27 06:54:27.443830972 (519h25m27.314914017s) [upload=455713]: Waiting for error group managing segments... Dec 27 06:54:27.727615936 (519h25m27.598698971s) [noise=true , address=145.79.227.3:4151 , piece_id=M263OMEQHOENHQ36XCPLPKG6OBQMBYIQRV5VAZC5X7JYB3I6O5QQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.735392764 (519h25m27.606475809s) [noise=true , address=145.79.227.15:4873 , piece_id=MUS34ISVIR67SYD6GT2V7MLEDJHRKH3CQQECXMFVMJC6YO7RBGYQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.736136841 (519h25m27.607219886s) [noise=true , address=145.79.227.36:4519 , piece_id=FYEMUY3GKOKZ6QQEQ23SE5WHJLFNUNEERYDJQHJBRW76ROLBA7WQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.742634093 (519h25m27.613717139s) [noise=true , address=145.79.227.38:4977 , piece_id=K7TTGWKUEOT2AC5QKYNBOMISX2ZLWKP53LYLPH6GIYXTNYHOZ3YA, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.743306494 (519h25m27.614389539s) [noise=true , address=145.79.227.50:4373 , piece_id=PG7D6WIGWLEUP357R2UNBGQHZ53LH3PE4XJRVCPMSSSIJJAX455A, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.744745183 (519h25m27.615828248s) [noise=true , address=145.79.227.12:4765 , piece_id=SXDZSFYNG3GQS2UJX47CYL7RVCJUVD2K6H6LYZNLGIZBSZKLVLQQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.745462709 (519h25m27.616545754s) [noise=true , address=145.79.227.19:4079 , piece_id=3MAJKX64LBC32QMBNOLP5CM4GLORBTANGJQ5ME7A4EC6WLH74VMA, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.745914390 (519h25m27.616997435s) [noise=true , address=145.79.227.42:4121 , piece_id=DNPVOU6BMRQ24PYZQCT6OSWXXW2X7SMGGA5OGLPL3AVCM7W733FQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.746268574 (519h25m27.617351609s) [noise=true , address=145.79.227.4:4186 , piece_id=FORBLNPBGKCCQIJIF4UJ2N7IMTV6KRLN4MTNNEIM6Q5A7Z34WM6A, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.765323108 (519h25m27.636406143s) [noise=true , address=145.79.227.48:4301 , piece_id=KPJMMTOAGUNQKNQSO5FRHRAARUUCZ4JBBP3EFE2LW6PUOCOUYTGA, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.775903355 (519h25m27.64698639s) [noise=true , address=145.79.227.47:4266 , piece_id=SQKABZSWUXJXXIYBXMDPXX3VNIHHYMESJHXHRNRR4TOQPR735OFQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:27.922976010 (519h25m27.794059045s) [noise=true , address=77.90.31.21:4878 , piece_id=JZAGUVLG6NJWOQK5ON7Z7FGOR7GYBJBOI6GKNR6CHIEJKYQZSNTQ, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:28.009083085 (519h25m27.88016612s) [noise=true , address=45.91.116.7:4010 , piece_id=UK73OUYF2D2QQU2J5TBYGB45PYZBDY7BE5I6XTQ3VAUQSSHACW2A, seg_pos={0 0}, upload=455713]: Done uploading piece. err: <nil> Dec 27 06:54:28.009104757 (519h25m27.880187802s) [seg_pos={0 0}, upload=455713]: Segment reached optimal threshold of 13 pieces. Dec 27 06:54:28.009276414 (519h25m27.880359459s) [noise=true , address=145.79.227.34:4905 , piece_id=6LX6ZHFFKNUMW5CLJIWLMPGLGMN6NVIZ5MEXQWULMMGHQHN3AC7A, seg_pos={0 0}, upload=455713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:28.009282746 (519h25m27.880365781s) [noise=true , address=145.79.227.34:4905 , piece_id=6LX6ZHFFKNUMW5CLJIWLMPGLGMN6NVIZ5MEXQWULMMGHQHN3AC7A, seg_pos={0 0}, upload=455713]: Long tail cancellation due to optimal threshold Dec 27 06:54:28.009387846 (519h25m27.880470891s) [noise=true , address=45.91.116.11:4120 , piece_id=FEOQVQXQQ7AZ63HKULHWYLOYATJRXWM4CK4DJ3QBOFXZIWAWPN3A, seg_pos={0 0}, upload=455713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:28.009393055 (519h25m27.880476101s) [noise=true , address=45.91.116.11:4120 , piece_id=FEOQVQXQQ7AZ63HKULHWYLOYATJRXWM4CK4DJ3QBOFXZIWAWPN3A, seg_pos={0 0}, upload=455713]: Long tail cancellation due to optimal threshold Dec 27 06:54:28.009646448 (519h25m27.880729493s) [noise=true , address=77.90.31.18:4808 , piece_id=5MGRYJQ2GC2VZJFPY4P5OODYX42BLJYPS7GZ7BFA7FWUU3SDJBUA, seg_pos={0 0}, upload=455713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:44266->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 06:54:28.009662238 (519h25m27.880745283s) [noise=true , address=77.90.31.18:4808 , piece_id=5MGRYJQ2GC2VZJFPY4P5OODYX42BLJYPS7GZ7BFA7FWUU3SDJBUA, seg_pos={0 0}, upload=455713]: Long tail cancellation due to optimal threshold Dec 27 06:54:28.009682857 (519h25m27.880765902s) [seg_pos={0 0}, upload=455713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:54:28.009773821 (519h25m27.880856856s) [upload=455713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:28.009779271 (519h25m27.880862306s) [upload=455713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:28.309163971 (519h25m28.180247006s) [upload=455713]: Done uploading segments... Dec 27 06:54:28.309190580 (519h25m28.180273616s) [upload=455713]: Upload finished. err: <nil> Duration: 30.003044276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 44.560321834s trace: https://jaeger.dev.storj.io/trace/7637d58679dfc2b FAIL (2025-12-27 06:54:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, Piece ID: 4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ) context deadline exceeded; piecestore: (Node ID: 1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, Piece ID: M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ) context deadline exceeded; piecestore: (Node ID: 12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, Piece ID: J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A) context deadline exceeded ======LOGS====== ec 27 06:23:01.300606186 (518h54m1.171689221s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300678243 (518h54m1.171761288s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300726415 (518h54m1.17180946s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300775528 (518h54m1.171858573s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300817849 (518h54m1.171900884s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300861301 (518h54m1.171944346s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300904174 (518h54m1.171987219s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.302497919 (518h54m1.173580954s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303015062 (518h54m1.174098117s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303683576 (518h54m1.174766621s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304243622 (518h54m1.175326668s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304804771 (518h54m1.175887816s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.305452815 (518h54m1.17653586s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.307203358 (518h54m1.178286393s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.308885732 (518h54m1.179968767s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.310641255 (518h54m1.1817243s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:02.661676879 (518h54m2.532759924s) [upload=455252]: Next returned nil segment Dec 27 06:23:02.661697388 (518h54m2.532780434s) [upload=455252]: Waiting for error group managing segments... Dec 27 06:23:03.006040636 (518h54m2.877123681s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.198726907 (518h54m3.069809952s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.209535790 (518h54m3.080618825s) [noise=true , address=145.79.227.29:4953 , piece_id=WTFPI4JRATOX4HGQO3NMPCKUS5JCVOQXUHMTUOIC4GOLJ2RBU7YA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.396894026 (518h54m3.267977071s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.444090588 (518h54m3.315173633s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.453033817 (518h54m3.324116862s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.461352988 (518h54m3.332436033s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.601936156 (518h54m3.473019201s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.076565583 (518h54m3.947648658s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.116623061 (518h54m3.987706106s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.121726017 (518h54m3.992809062s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.132649729 (518h54m4.003732784s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.139015891 (518h54m4.010098936s) [noise=true , address=145.79.227.43:4172 , piece_id=UYDTZF562756XZFBJO3FKVWZ57HBZ7W33OUOUJHECY4HKY3CCWZA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169389909 (518h54m4.040472954s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169423092 (518h54m4.040506137s) [seg_pos={0 0}, upload=455252]: Segment reached optimal threshold of 14 pieces. Dec 27 06:23:04.169650115 (518h54m4.04073316s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection Dec 27 06:23:04.169661276 (518h54m4.040744321s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169734896 (518h54m4.040817961s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169756528 (518h54m4.040839583s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169892065 (518h54m4.04097512s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126JXfJvuad1JedXjgEJuA12awNtwHBYNZhHSJoei2ZYTrccDmA): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169902125 (518h54m4.04098518s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170085444 (518h54m4.041168489s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.170094631 (518h54m4.041177676s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170205322 (518h54m4.041288367s) [seg_pos={0 0}, upload=455252]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:23:04.170345309 (518h54m4.041428364s) [upload=455252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:23:04.170352803 (518h54m4.041435848s) [upload=455252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:23:04.479479019 (518h54m4.350562065s) [upload=455252]: Done uploading segments... Dec 27 06:23:04.479517072 (518h54m4.350600107s) [upload=455252]: Upload finished. err: <nil> Duration: 44.560357251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................X.....X.........X.....X..........] duration: 3.435512625s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:54:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435541419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......X...........XX.XXXXX......X.XX..XXX...X.XX........X.X.] duration: 5.868097733s trace: https://jaeger.dev.storj.io/trace/540e8d6fd5071b0d FAIL (2025-12-27 06:53:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 06:52:37.001693179 (519h23m36.872776224s) [upload=455686]: Starting upload Dec 27 06:52:37.001768763 (519h23m36.872851798s) [upload=455686]: Done starting upload Dec 27 06:52:37.001892629 (519h23m36.872975664s) [upload=455686]: Uploading segments... Dec 27 06:52:37.002181880 (519h23m36.873264925s) [upload=455686]: Got next segment. Inline: false Dec 27 06:52:37.002241463 (519h23m36.873324508s) [upload=455686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:52:37.002259167 (519h23m36.873342202s) [upload=455686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:52:37.248624926 (519h23m37.119707961s) [seg_pos={0 0}, upload=455686]: Begin upload segment... Dec 27 06:52:37.248669130 (519h23m37.119752165s) [seg_pos={0 0}, upload=455686]: Done begin upload segment. Dec 27 06:52:37.248692285 (519h23m37.11977532s) [noise=true , address=45.91.116.11:4120 , piece_id=BU5IGSHU7JB6CL4WLW6KYBPARDRX2RM2NECAHYLLYDT5P5HVPRPQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.248777057 (519h23m37.119860102s) [noise=true , address=45.91.116.14:4189 , piece_id=MDCNAML26IBZGVFILSTORQ4LUB4SVVENHIUQCEQVHQ4V5MUMIAGQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.249572420 (519h23m37.120655465s) [noise=true , address=145.79.227.35:4481 , piece_id=HIWR54JBKVUUBHTLYU3EKDPCQ5WXANOTPETTY6SKZEBNTKLOX25A, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.249663504 (519h23m37.120746539s) [noise=true , address=145.79.227.50:4373 , piece_id=PPK4VGY7AKQZE6ZUO73WQIDPTZ47KXY3FWU6FBWPAPRCOTXKNYCQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250047635 (519h23m37.121130671s) [noise=true , address=145.79.227.19:4079 , piece_id=CUHZZRPESLG5HVM373V6PRXCQN3KNFZGP6PXL6UJFZVGH4XPFYVA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250135092 (519h23m37.121218128s) [noise=true , address=145.79.227.40:4049 , piece_id=KPFH7XRLNUKRZW3FJVZBVPPKNNOM33PKQTQ4KXEUQ5PYOTLQLHUA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250341546 (519h23m37.121424581s) [noise=true , address=145.79.227.31:4797 , piece_id=JIFUQE7TDJ24JLY2IJQ4UMBRZB7OLUOHTA2OF7DUPX4UC3CEYTYA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250502301 (519h23m37.121585346s) [noise=true , address=145.79.227.9:4366 , piece_id=7CV4J3EUYP4HEMQWWOGKQSM3FC75V5B233T7V3FW6Q4AGIFNCPXA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250691502 (519h23m37.121774537s) [noise=true , address=145.79.227.11:4732 , piece_id=ESYFUGYG7HQKYXBR5PO2WW2QKPDWRK67K3UFM5YV7YF4V57AI4QA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250874079 (519h23m37.121957114s) [noise=true , address=45.91.116.9:4261 , piece_id=Z2ITSL4OSMR6HSTEOVCNOLN73UTK7Q6UYDAPZWZF5HW6OSZPMUKQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250957368 (519h23m37.122040403s) [noise=true , address=145.79.227.44:4193 , piece_id=AH6PHSZQTV7ZY3XQ4TRGVOFQVWOWCB47YTP5VCDB4OJWTHTF2CLA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251241018 (519h23m37.122324053s) [noise=true , address=145.79.227.51:4411 , piece_id=X4NU4P77U5SKYJNYXVZ4CNBJXZ3HTDDMVB5UO3CTNA5JN2QT34QA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251306873 (519h23m37.122389908s) [noise=true , address=77.90.31.19:4844 , piece_id=YPKF5BF27QBGQPYSRNK5C7V6FB6MDMOEJE6FUWODROLCIANDYKOQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251567500 (519h23m37.122650535s) [noise=true , address=145.79.227.15:4873 , piece_id=YIGV6CZZQ47IAG2OVWIMQRII5HUKT6W6N3JDQ5JWRDTU5WH7I52Q, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251791867 (519h23m37.122874902s) [noise=true , address=145.79.227.29:4942 , piece_id=FMAK5TQ2CJHHBGWS7BSFORLGHDSESHIVJ3UTQEDFCJLJ3XZTVE4Q, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.252080737 (519h23m37.123163772s) [noise=true , address=145.79.227.38:4978 , piece_id=2J74MIUP4BTZWSTST555YKOPK5FMRAEV6DGKL5M2T2J4ZREABM2A, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:38.963767391 (519h23m38.834850436s) [upload=455686]: Next returned nil segment Dec 27 06:52:38.963803089 (519h23m38.834886134s) [upload=455686]: Waiting for error group managing segments... Dec 27 06:52:39.255815058 (519h23m39.126898103s) [noise=true , address=145.79.227.38:4978 , piece_id=2J74MIUP4BTZWSTST555YKOPK5FMRAEV6DGKL5M2T2J4ZREABM2A, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.256339937 (519h23m39.127422972s) [noise=true , address=145.79.227.9:4366 , piece_id=7CV4J3EUYP4HEMQWWOGKQSM3FC75V5B233T7V3FW6Q4AGIFNCPXA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.256818799 (519h23m39.127901844s) [noise=true , address=145.79.227.51:4411 , piece_id=X4NU4P77U5SKYJNYXVZ4CNBJXZ3HTDDMVB5UO3CTNA5JN2QT34QA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.260221799 (519h23m39.131304844s) [noise=true , address=145.79.227.31:4797 , piece_id=JIFUQE7TDJ24JLY2IJQ4UMBRZB7OLUOHTA2OF7DUPX4UC3CEYTYA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.260547400 (519h23m39.131630435s) [noise=true , address=145.79.227.50:4373 , piece_id=PPK4VGY7AKQZE6ZUO73WQIDPTZ47KXY3FWU6FBWPAPRCOTXKNYCQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.262516549 (519h23m39.133599594s) [noise=true , address=145.79.227.29:4942 , piece_id=FMAK5TQ2CJHHBGWS7BSFORLGHDSESHIVJ3UTQEDFCJLJ3XZTVE4Q, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.263116321 (519h23m39.134199356s) [noise=true , address=145.79.227.11:4732 , piece_id=ESYFUGYG7HQKYXBR5PO2WW2QKPDWRK67K3UFM5YV7YF4V57AI4QA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.270918839 (519h23m39.142001874s) [noise=true , address=145.79.227.19:4079 , piece_id=CUHZZRPESLG5HVM373V6PRXCQN3KNFZGP6PXL6UJFZVGH4XPFYVA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.276563407 (519h23m39.147646442s) [noise=true , address=145.79.227.35:4481 , piece_id=HIWR54JBKVUUBHTLYU3EKDPCQ5WXANOTPETTY6SKZEBNTKLOX25A, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.277197424 (519h23m39.148280469s) [noise=true , address=145.79.227.15:4873 , piece_id=YIGV6CZZQ47IAG2OVWIMQRII5HUKT6W6N3JDQ5JWRDTU5WH7I52Q, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.436310637 (519h23m39.307393672s) [noise=true , address=77.90.31.19:4844 , piece_id=YPKF5BF27QBGQPYSRNK5C7V6FB6MDMOEJE6FUWODROLCIANDYKOQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.589597747 (519h23m39.460680792s) [noise=true , address=45.91.116.9:4261 , piece_id=Z2ITSL4OSMR6HSTEOVCNOLN73UTK7Q6UYDAPZWZF5HW6OSZPMUKQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.611094932 (519h23m39.482177967s) [noise=true , address=45.91.116.14:4189 , piece_id=MDCNAML26IBZGVFILSTORQ4LUB4SVVENHIUQCEQVHQ4V5MUMIAGQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.611119228 (519h23m39.482202263s) [seg_pos={0 0}, upload=455686]: Segment reached optimal threshold of 13 pieces. Dec 27 06:52:39.611368803 (519h23m39.482451848s) [noise=true , address=45.91.116.11:4120 , piece_id=BU5IGSHU7JB6CL4WLW6KYBPARDRX2RM2NECAHYLLYDT5P5HVPRPQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 06:52:39.611376408 (519h23m39.482459443s) [noise=true , address=45.91.116.11:4120 , piece_id=BU5IGSHU7JB6CL4WLW6KYBPARDRX2RM2NECAHYLLYDT5P5HVPRPQ, seg_pos={0 0}, upload=455686]: Long tail cancellation due to optimal threshold Dec 27 06:52:39.611569085 (519h23m39.48265213s) [noise=true , address=145.79.227.40:4049 , piece_id=KPFH7XRLNUKRZW3FJVZBVPPKNNOM33PKQTQ4KXEUQ5PYOTLQLHUA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 27 06:52:39.611587600 (519h23m39.482670635s) [noise=true , address=145.79.227.40:4049 , piece_id=KPFH7XRLNUKRZW3FJVZBVPPKNNOM33PKQTQ4KXEUQ5PYOTLQLHUA, seg_pos={0 0}, upload=455686]: Long tail cancellation due to optimal threshold Dec 27 06:52:39.611766510 (519h23m39.482849555s) [noise=true , address=145.79.227.44:4193 , piece_id=AH6PHSZQTV7ZY3XQ4TRGVOFQVWOWCB47YTP5VCDB4OJWTHTF2CLA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 06:52:39.611777612 (519h23m39.482860647s) [noise=true , address=145.79.227.44:4193 , piece_id=AH6PHSZQTV7ZY3XQ4TRGVOFQVWOWCB47YTP5VCDB4OJWTHTF2CLA, seg_pos={0 0}, upload=455686]: Long tail cancellation due to optimal threshold Dec 27 06:52:39.611802739 (519h23m39.482885774s) [seg_pos={0 0}, upload=455686]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:52:39.611883634 (519h23m39.482966669s) [upload=455686]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:52:39.611889234 (519h23m39.482972269s) [upload=455686]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:52:39.909169455 (519h23m39.78025252s) [upload=455686]: Done uploading segments... Dec 27 06:52:39.909193782 (519h23m39.780276847s) [upload=455686]: Upload finished. err: <nil> Duration: 5.868117661s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................X.....X.........X.....X.........] duration: 3.480973751s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:53:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481005281s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................X.....X.........X.....X........] duration: 3.456980912s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:53:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457010347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 35.20492485s trace: https://jaeger.dev.storj.io/trace/7637d58679dfc2b FAIL (2025-12-27 06:53:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, Piece ID: 4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ) context deadline exceeded; piecestore: (Node ID: 1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, Piece ID: M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ) context deadline exceeded; piecestore: (Node ID: 12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, Piece ID: J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A) context deadline exceeded ======LOGS====== ec 27 06:23:01.300606186 (518h54m1.171689221s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300678243 (518h54m1.171761288s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300726415 (518h54m1.17180946s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300775528 (518h54m1.171858573s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300817849 (518h54m1.171900884s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300861301 (518h54m1.171944346s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300904174 (518h54m1.171987219s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.302497919 (518h54m1.173580954s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303015062 (518h54m1.174098117s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303683576 (518h54m1.174766621s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304243622 (518h54m1.175326668s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304804771 (518h54m1.175887816s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.305452815 (518h54m1.17653586s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.307203358 (518h54m1.178286393s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.308885732 (518h54m1.179968767s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.310641255 (518h54m1.1817243s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:02.661676879 (518h54m2.532759924s) [upload=455252]: Next returned nil segment Dec 27 06:23:02.661697388 (518h54m2.532780434s) [upload=455252]: Waiting for error group managing segments... Dec 27 06:23:03.006040636 (518h54m2.877123681s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.198726907 (518h54m3.069809952s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.209535790 (518h54m3.080618825s) [noise=true , address=145.79.227.29:4953 , piece_id=WTFPI4JRATOX4HGQO3NMPCKUS5JCVOQXUHMTUOIC4GOLJ2RBU7YA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.396894026 (518h54m3.267977071s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.444090588 (518h54m3.315173633s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.453033817 (518h54m3.324116862s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.461352988 (518h54m3.332436033s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.601936156 (518h54m3.473019201s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.076565583 (518h54m3.947648658s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.116623061 (518h54m3.987706106s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.121726017 (518h54m3.992809062s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.132649729 (518h54m4.003732784s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.139015891 (518h54m4.010098936s) [noise=true , address=145.79.227.43:4172 , piece_id=UYDTZF562756XZFBJO3FKVWZ57HBZ7W33OUOUJHECY4HKY3CCWZA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169389909 (518h54m4.040472954s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169423092 (518h54m4.040506137s) [seg_pos={0 0}, upload=455252]: Segment reached optimal threshold of 14 pieces. Dec 27 06:23:04.169650115 (518h54m4.04073316s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection Dec 27 06:23:04.169661276 (518h54m4.040744321s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169734896 (518h54m4.040817961s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169756528 (518h54m4.040839583s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169892065 (518h54m4.04097512s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126JXfJvuad1JedXjgEJuA12awNtwHBYNZhHSJoei2ZYTrccDmA): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169902125 (518h54m4.04098518s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170085444 (518h54m4.041168489s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.170094631 (518h54m4.041177676s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170205322 (518h54m4.041288367s) [seg_pos={0 0}, upload=455252]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:23:04.170345309 (518h54m4.041428364s) [upload=455252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:23:04.170352803 (518h54m4.041435848s) [upload=455252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:23:04.479479019 (518h54m4.350562065s) [upload=455252]: Done uploading segments... Dec 27 06:23:04.479517072 (518h54m4.350600107s) [upload=455252]: Upload finished. err: <nil> Duration: 35.204942103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......X...........XX.XXXXX......X.XX..XXX...X.XX........X.X] duration: 30.002511567s trace: https://jaeger.dev.storj.io/trace/540e8d6fd5071b0d FAIL (2025-12-27 06:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 06:52:37.001693179 (519h23m36.872776224s) [upload=455686]: Starting upload Dec 27 06:52:37.001768763 (519h23m36.872851798s) [upload=455686]: Done starting upload Dec 27 06:52:37.001892629 (519h23m36.872975664s) [upload=455686]: Uploading segments... Dec 27 06:52:37.002181880 (519h23m36.873264925s) [upload=455686]: Got next segment. Inline: false Dec 27 06:52:37.002241463 (519h23m36.873324508s) [upload=455686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:52:37.002259167 (519h23m36.873342202s) [upload=455686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:52:37.248624926 (519h23m37.119707961s) [seg_pos={0 0}, upload=455686]: Begin upload segment... Dec 27 06:52:37.248669130 (519h23m37.119752165s) [seg_pos={0 0}, upload=455686]: Done begin upload segment. Dec 27 06:52:37.248692285 (519h23m37.11977532s) [noise=true , address=45.91.116.11:4120 , piece_id=BU5IGSHU7JB6CL4WLW6KYBPARDRX2RM2NECAHYLLYDT5P5HVPRPQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.248777057 (519h23m37.119860102s) [noise=true , address=45.91.116.14:4189 , piece_id=MDCNAML26IBZGVFILSTORQ4LUB4SVVENHIUQCEQVHQ4V5MUMIAGQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.249572420 (519h23m37.120655465s) [noise=true , address=145.79.227.35:4481 , piece_id=HIWR54JBKVUUBHTLYU3EKDPCQ5WXANOTPETTY6SKZEBNTKLOX25A, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.249663504 (519h23m37.120746539s) [noise=true , address=145.79.227.50:4373 , piece_id=PPK4VGY7AKQZE6ZUO73WQIDPTZ47KXY3FWU6FBWPAPRCOTXKNYCQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250047635 (519h23m37.121130671s) [noise=true , address=145.79.227.19:4079 , piece_id=CUHZZRPESLG5HVM373V6PRXCQN3KNFZGP6PXL6UJFZVGH4XPFYVA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250135092 (519h23m37.121218128s) [noise=true , address=145.79.227.40:4049 , piece_id=KPFH7XRLNUKRZW3FJVZBVPPKNNOM33PKQTQ4KXEUQ5PYOTLQLHUA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250341546 (519h23m37.121424581s) [noise=true , address=145.79.227.31:4797 , piece_id=JIFUQE7TDJ24JLY2IJQ4UMBRZB7OLUOHTA2OF7DUPX4UC3CEYTYA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250502301 (519h23m37.121585346s) [noise=true , address=145.79.227.9:4366 , piece_id=7CV4J3EUYP4HEMQWWOGKQSM3FC75V5B233T7V3FW6Q4AGIFNCPXA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250691502 (519h23m37.121774537s) [noise=true , address=145.79.227.11:4732 , piece_id=ESYFUGYG7HQKYXBR5PO2WW2QKPDWRK67K3UFM5YV7YF4V57AI4QA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250874079 (519h23m37.121957114s) [noise=true , address=45.91.116.9:4261 , piece_id=Z2ITSL4OSMR6HSTEOVCNOLN73UTK7Q6UYDAPZWZF5HW6OSZPMUKQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.250957368 (519h23m37.122040403s) [noise=true , address=145.79.227.44:4193 , piece_id=AH6PHSZQTV7ZY3XQ4TRGVOFQVWOWCB47YTP5VCDB4OJWTHTF2CLA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251241018 (519h23m37.122324053s) [noise=true , address=145.79.227.51:4411 , piece_id=X4NU4P77U5SKYJNYXVZ4CNBJXZ3HTDDMVB5UO3CTNA5JN2QT34QA, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251306873 (519h23m37.122389908s) [noise=true , address=77.90.31.19:4844 , piece_id=YPKF5BF27QBGQPYSRNK5C7V6FB6MDMOEJE6FUWODROLCIANDYKOQ, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251567500 (519h23m37.122650535s) [noise=true , address=145.79.227.15:4873 , piece_id=YIGV6CZZQ47IAG2OVWIMQRII5HUKT6W6N3JDQ5JWRDTU5WH7I52Q, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.251791867 (519h23m37.122874902s) [noise=true , address=145.79.227.29:4942 , piece_id=FMAK5TQ2CJHHBGWS7BSFORLGHDSESHIVJ3UTQEDFCJLJ3XZTVE4Q, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:37.252080737 (519h23m37.123163772s) [noise=true , address=145.79.227.38:4978 , piece_id=2J74MIUP4BTZWSTST555YKOPK5FMRAEV6DGKL5M2T2J4ZREABM2A, seg_pos={0 0}, upload=455686]: Uploading piece... Dec 27 06:52:38.963767391 (519h23m38.834850436s) [upload=455686]: Next returned nil segment Dec 27 06:52:38.963803089 (519h23m38.834886134s) [upload=455686]: Waiting for error group managing segments... Dec 27 06:52:39.255815058 (519h23m39.126898103s) [noise=true , address=145.79.227.38:4978 , piece_id=2J74MIUP4BTZWSTST555YKOPK5FMRAEV6DGKL5M2T2J4ZREABM2A, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.256339937 (519h23m39.127422972s) [noise=true , address=145.79.227.9:4366 , piece_id=7CV4J3EUYP4HEMQWWOGKQSM3FC75V5B233T7V3FW6Q4AGIFNCPXA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.256818799 (519h23m39.127901844s) [noise=true , address=145.79.227.51:4411 , piece_id=X4NU4P77U5SKYJNYXVZ4CNBJXZ3HTDDMVB5UO3CTNA5JN2QT34QA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.260221799 (519h23m39.131304844s) [noise=true , address=145.79.227.31:4797 , piece_id=JIFUQE7TDJ24JLY2IJQ4UMBRZB7OLUOHTA2OF7DUPX4UC3CEYTYA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.260547400 (519h23m39.131630435s) [noise=true , address=145.79.227.50:4373 , piece_id=PPK4VGY7AKQZE6ZUO73WQIDPTZ47KXY3FWU6FBWPAPRCOTXKNYCQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.262516549 (519h23m39.133599594s) [noise=true , address=145.79.227.29:4942 , piece_id=FMAK5TQ2CJHHBGWS7BSFORLGHDSESHIVJ3UTQEDFCJLJ3XZTVE4Q, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.263116321 (519h23m39.134199356s) [noise=true , address=145.79.227.11:4732 , piece_id=ESYFUGYG7HQKYXBR5PO2WW2QKPDWRK67K3UFM5YV7YF4V57AI4QA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.270918839 (519h23m39.142001874s) [noise=true , address=145.79.227.19:4079 , piece_id=CUHZZRPESLG5HVM373V6PRXCQN3KNFZGP6PXL6UJFZVGH4XPFYVA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.276563407 (519h23m39.147646442s) [noise=true , address=145.79.227.35:4481 , piece_id=HIWR54JBKVUUBHTLYU3EKDPCQ5WXANOTPETTY6SKZEBNTKLOX25A, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.277197424 (519h23m39.148280469s) [noise=true , address=145.79.227.15:4873 , piece_id=YIGV6CZZQ47IAG2OVWIMQRII5HUKT6W6N3JDQ5JWRDTU5WH7I52Q, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.436310637 (519h23m39.307393672s) [noise=true , address=77.90.31.19:4844 , piece_id=YPKF5BF27QBGQPYSRNK5C7V6FB6MDMOEJE6FUWODROLCIANDYKOQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.589597747 (519h23m39.460680792s) [noise=true , address=45.91.116.9:4261 , piece_id=Z2ITSL4OSMR6HSTEOVCNOLN73UTK7Q6UYDAPZWZF5HW6OSZPMUKQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.611094932 (519h23m39.482177967s) [noise=true , address=45.91.116.14:4189 , piece_id=MDCNAML26IBZGVFILSTORQ4LUB4SVVENHIUQCEQVHQ4V5MUMIAGQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: <nil> Dec 27 06:52:39.611119228 (519h23m39.482202263s) [seg_pos={0 0}, upload=455686]: Segment reached optimal threshold of 13 pieces. Dec 27 06:52:39.611368803 (519h23m39.482451848s) [noise=true , address=45.91.116.11:4120 , piece_id=BU5IGSHU7JB6CL4WLW6KYBPARDRX2RM2NECAHYLLYDT5P5HVPRPQ, seg_pos={0 0}, upload=455686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 06:52:39.611376408 (519h23m39.482459443s) [noise=true , address=45.91.116.11:4120 , piece_id=BU5IGSHU7JB6CL4WLW6KYBPARDRX2RM2NECAHYLLYDT5P5HVPRPQ, seg_pos={0 0}, upload=455686]: Long tail cancellation due to optimal threshold Dec 27 06:52:39.611569085 (519h23m39.48265213s) [noise=true , address=145.79.227.40:4049 , piece_id=KPFH7XRLNUKRZW3FJVZBVPPKNNOM33PKQTQ4KXEUQ5PYOTLQLHUA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 27 06:52:39.611587600 (519h23m39.482670635s) [noise=true , address=145.79.227.40:4049 , piece_id=KPFH7XRLNUKRZW3FJVZBVPPKNNOM33PKQTQ4KXEUQ5PYOTLQLHUA, seg_pos={0 0}, upload=455686]: Long tail cancellation due to optimal threshold Dec 27 06:52:39.611766510 (519h23m39.482849555s) [noise=true , address=145.79.227.44:4193 , piece_id=AH6PHSZQTV7ZY3XQ4TRGVOFQVWOWCB47YTP5VCDB4OJWTHTF2CLA, seg_pos={0 0}, upload=455686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 06:52:39.611777612 (519h23m39.482860647s) [noise=true , address=145.79.227.44:4193 , piece_id=AH6PHSZQTV7ZY3XQ4TRGVOFQVWOWCB47YTP5VCDB4OJWTHTF2CLA, seg_pos={0 0}, upload=455686]: Long tail cancellation due to optimal threshold Dec 27 06:52:39.611802739 (519h23m39.482885774s) [seg_pos={0 0}, upload=455686]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:52:39.611883634 (519h23m39.482966669s) [upload=455686]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:52:39.611889234 (519h23m39.482972269s) [upload=455686]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:52:39.909169455 (519h23m39.78025252s) [upload=455686]: Done uploading segments... Dec 27 06:52:39.909193782 (519h23m39.780276847s) [upload=455686]: Upload finished. err: <nil> Duration: 30.002536063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 36.972152012s trace: https://jaeger.dev.storj.io/trace/7637d58679dfc2b FAIL (2025-12-27 06:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, Piece ID: 4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ) context deadline exceeded; piecestore: (Node ID: 1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, Piece ID: M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ) context deadline exceeded; piecestore: (Node ID: 12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, Piece ID: J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A) context deadline exceeded ======LOGS====== ec 27 06:23:01.300606186 (518h54m1.171689221s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300678243 (518h54m1.171761288s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300726415 (518h54m1.17180946s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300775528 (518h54m1.171858573s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300817849 (518h54m1.171900884s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300861301 (518h54m1.171944346s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300904174 (518h54m1.171987219s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.302497919 (518h54m1.173580954s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303015062 (518h54m1.174098117s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303683576 (518h54m1.174766621s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304243622 (518h54m1.175326668s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304804771 (518h54m1.175887816s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.305452815 (518h54m1.17653586s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.307203358 (518h54m1.178286393s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.308885732 (518h54m1.179968767s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.310641255 (518h54m1.1817243s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:02.661676879 (518h54m2.532759924s) [upload=455252]: Next returned nil segment Dec 27 06:23:02.661697388 (518h54m2.532780434s) [upload=455252]: Waiting for error group managing segments... Dec 27 06:23:03.006040636 (518h54m2.877123681s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.198726907 (518h54m3.069809952s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.209535790 (518h54m3.080618825s) [noise=true , address=145.79.227.29:4953 , piece_id=WTFPI4JRATOX4HGQO3NMPCKUS5JCVOQXUHMTUOIC4GOLJ2RBU7YA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.396894026 (518h54m3.267977071s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.444090588 (518h54m3.315173633s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.453033817 (518h54m3.324116862s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.461352988 (518h54m3.332436033s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.601936156 (518h54m3.473019201s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.076565583 (518h54m3.947648658s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.116623061 (518h54m3.987706106s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.121726017 (518h54m3.992809062s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.132649729 (518h54m4.003732784s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.139015891 (518h54m4.010098936s) [noise=true , address=145.79.227.43:4172 , piece_id=UYDTZF562756XZFBJO3FKVWZ57HBZ7W33OUOUJHECY4HKY3CCWZA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169389909 (518h54m4.040472954s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169423092 (518h54m4.040506137s) [seg_pos={0 0}, upload=455252]: Segment reached optimal threshold of 14 pieces. Dec 27 06:23:04.169650115 (518h54m4.04073316s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection Dec 27 06:23:04.169661276 (518h54m4.040744321s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169734896 (518h54m4.040817961s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169756528 (518h54m4.040839583s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169892065 (518h54m4.04097512s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126JXfJvuad1JedXjgEJuA12awNtwHBYNZhHSJoei2ZYTrccDmA): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169902125 (518h54m4.04098518s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170085444 (518h54m4.041168489s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.170094631 (518h54m4.041177676s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170205322 (518h54m4.041288367s) [seg_pos={0 0}, upload=455252]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:23:04.170345309 (518h54m4.041428364s) [upload=455252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:23:04.170352803 (518h54m4.041435848s) [upload=455252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:23:04.479479019 (518h54m4.350562065s) [upload=455252]: Done uploading segments... Dec 27 06:23:04.479517072 (518h54m4.350600107s) [upload=455252]: Upload finished. err: <nil> Duration: 36.972170738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................X.....X.........X.....X.......] duration: 3.507866559s trace: https://jaeger.dev.storj.io/trace/3a40464c1c1fd3a2 FAIL (2025-12-27 06:52:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.507890386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X...........XX.XXXXX......X.XX..XXX...X.XX........X.] duration: 5.727395515s trace: https://jaeger.dev.storj.io/trace/28ecb31ed3cb3875 FAIL (2025-12-27 06:52:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: M5MPXLYGQTZTPLHMCKIYZOWUWG6BK7QMM2CL6QMYZQCGJWFAWSIA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: MXBE3O6WCZYOAQCTPNSJFL4UKWF564XOKHQRRVJGRVNC3S5BKFYA) context deadline exceeded ======LOGS====== Dec 27 06:51:35.701892242 (519h22m35.572975278s) [upload=455668]: Starting upload Dec 27 06:51:35.701954060 (519h22m35.573037095s) [upload=455668]: Done starting upload Dec 27 06:51:35.702123633 (519h22m35.573206678s) [upload=455668]: Uploading segments... Dec 27 06:51:35.702237990 (519h22m35.573321035s) [upload=455668]: Got next segment. Inline: false Dec 27 06:51:35.702279781 (519h22m35.573362816s) [upload=455668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:51:35.702285101 (519h22m35.573368136s) [upload=455668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:51:35.945197574 (519h22m35.816280609s) [seg_pos={0 0}, upload=455668]: Begin upload segment... Dec 27 06:51:35.945244723 (519h22m35.816327758s) [seg_pos={0 0}, upload=455668]: Done begin upload segment. Dec 27 06:51:35.945276845 (519h22m35.81635988s) [noise=true , address=145.79.227.13:4804 , piece_id=PEWVMEWQ2K3UH7LJQAHIVOT6LM5DLX6WMZTJC6T65AM3TZREC5OA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945307914 (519h22m35.816390949s) [noise=true , address=145.79.227.11:4731 , piece_id=YTCFQXWP5ERFXT4LTIXLNOEMGFGIO2Y5JQVOCENXM3D5NGLM7PWQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945645106 (519h22m35.816728141s) [noise=true , address=145.79.227.48:4302 , piece_id=B4YYGB7SQALIJLQB3M2AY6D7DWGKKVXGYO2DMW6YFW7KCYLKPELQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945655215 (519h22m35.81673825s) [noise=true , address=145.79.227.25:4617 , piece_id=RWWCSOURTNKK2SE5PLHSOKM4IPIY7EFJ2Q6IMEDMMOYV7R6QFFCQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945929858 (519h22m35.817012903s) [noise=true , address=45.91.116.8:4048 , piece_id=GP4RTD5A54JP6JTHHQRX7VCHMTEG2Z473MBIWDP26XIV4TFGELSA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945949576 (519h22m35.817032611s) [noise=true , address=145.79.227.15:4874 , piece_id=FMGEYUJWDWODE5DE3HQ4JOK5REL4GFLEFCRUUYPT3N2FKEUVFNQQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946184673 (519h22m35.817267708s) [noise=true , address=145.79.227.37:4567 , piece_id=NVAZONYQXHP74WDR5LP2TTYE7RYMVSKNXHT4HS3IWCFRUUIRPZHA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946202297 (519h22m35.817285342s) [noise=true , address=77.90.31.17:4773 , piece_id=EANLWZ4BCHIE6VVHK2K2BOA7F2SOVSTKCTJOFRJ3X6L55HSPNTKQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946424901 (519h22m35.817507936s) [noise=true , address=145.79.227.29:4943 , piece_id=HNHZUPVW6GB5PSIJOYHJLMRWLK433GPOZTEGA2D35ONBQOZMJJIA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946451482 (519h22m35.817534517s) [noise=true , address=145.79.227.3:4151 , piece_id=TFBW5LBDUYN4FSD6CMHCOO32C4XT4NZ7UGHRDABOCCYWIOW4T3PQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946662954 (519h22m35.817745989s) [noise=true , address=77.90.31.11:4555 , piece_id=MXBE3O6WCZYOAQCTPNSJFL4UKWF564XOKHQRRVJGRVNC3S5BKFYA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946684155 (519h22m35.8177672s) [noise=true , address=145.79.227.46:4602 , piece_id=ABZ2HDEK5FZYWM6SGWMUFUJI2YQYMVENKUB3IEIG7SRTEPW3UEHQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946916657 (519h22m35.817999692s) [noise=true , address=145.79.227.2:4113 , piece_id=ZL4N6NAWNP4A2SZQQGVWV4G7U3RETFD2U275OFD4UCJSYDUTXZJQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946951453 (519h22m35.818034498s) [noise=true , address=145.79.227.19:4079 , piece_id=KT6VYJUOAXAO6N35TF767DNL6WZKLYMQNBFWAE6I74QO3DJABUCA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.947187092 (519h22m35.818270127s) [noise=true , address=45.91.116.5:4300 , piece_id=M5MPXLYGQTZTPLHMCKIYZOWUWG6BK7QMM2CL6QMYZQCGJWFAWSIA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.948002525 (519h22m35.81908556s) [noise=true , address=45.91.116.9:4261 , piece_id=LXNAYAZC2SEGM5RVU3UD755AEDJ3TFWJ4P2YCPM5C3UYEEHD2K7Q, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:37.565621726 (519h22m37.436704771s) [upload=455668]: Next returned nil segment Dec 27 06:51:37.565635062 (519h22m37.436718097s) [upload=455668]: Waiting for error group managing segments... Dec 27 06:51:37.950016961 (519h22m37.821100006s) [noise=true , address=145.79.227.3:4151 , piece_id=TFBW5LBDUYN4FSD6CMHCOO32C4XT4NZ7UGHRDABOCCYWIOW4T3PQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.951331214 (519h22m37.822414259s) [noise=true , address=145.79.227.2:4113 , piece_id=ZL4N6NAWNP4A2SZQQGVWV4G7U3RETFD2U275OFD4UCJSYDUTXZJQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.952017250 (519h22m37.823100296s) [noise=true , address=145.79.227.37:4567 , piece_id=NVAZONYQXHP74WDR5LP2TTYE7RYMVSKNXHT4HS3IWCFRUUIRPZHA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.954793497 (519h22m37.825876542s) [noise=true , address=145.79.227.13:4804 , piece_id=PEWVMEWQ2K3UH7LJQAHIVOT6LM5DLX6WMZTJC6T65AM3TZREC5OA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.957724708 (519h22m37.828807754s) [noise=true , address=145.79.227.46:4602 , piece_id=ABZ2HDEK5FZYWM6SGWMUFUJI2YQYMVENKUB3IEIG7SRTEPW3UEHQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.957866007 (519h22m37.828949053s) [noise=true , address=145.79.227.11:4731 , piece_id=YTCFQXWP5ERFXT4LTIXLNOEMGFGIO2Y5JQVOCENXM3D5NGLM7PWQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.958337946 (519h22m37.829420981s) [noise=true , address=145.79.227.25:4617 , piece_id=RWWCSOURTNKK2SE5PLHSOKM4IPIY7EFJ2Q6IMEDMMOYV7R6QFFCQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.962947544 (519h22m37.834030589s) [noise=true , address=145.79.227.19:4079 , piece_id=KT6VYJUOAXAO6N35TF767DNL6WZKLYMQNBFWAE6I74QO3DJABUCA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.964975807 (519h22m37.836058842s) [noise=true , address=145.79.227.48:4302 , piece_id=B4YYGB7SQALIJLQB3M2AY6D7DWGKKVXGYO2DMW6YFW7KCYLKPELQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.968287202 (519h22m37.839370237s) [noise=true , address=145.79.227.15:4874 , piece_id=FMGEYUJWDWODE5DE3HQ4JOK5REL4GFLEFCRUUYPT3N2FKEUVFNQQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.976078558 (519h22m37.847161603s) [noise=true , address=145.79.227.29:4943 , piece_id=HNHZUPVW6GB5PSIJOYHJLMRWLK433GPOZTEGA2D35ONBQOZMJJIA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:38.948145758 (519h22m38.819228803s) [noise=true , address=77.90.31.11:4555 , piece_id=MXBE3O6WCZYOAQCTPNSJFL4UKWF564XOKHQRRVJGRVNC3S5BKFYA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:38.966636899 (519h22m38.837719934s) [noise=true , address=45.91.116.5:4300 , piece_id=M5MPXLYGQTZTPLHMCKIYZOWUWG6BK7QMM2CL6QMYZQCGJWFAWSIA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:38.966658028 (519h22m38.837741063s) [seg_pos={0 0}, upload=455668]: Segment reached optimal threshold of 13 pieces. Dec 27 06:51:38.966866135 (519h22m38.83794917s) [noise=true , address=77.90.31.17:4773 , piece_id=EANLWZ4BCHIE6VVHK2K2BOA7F2SOVSTKCTJOFRJ3X6L55HSPNTKQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 27 06:51:38.966874371 (519h22m38.837957406s) [noise=true , address=77.90.31.17:4773 , piece_id=EANLWZ4BCHIE6VVHK2K2BOA7F2SOVSTKCTJOFRJ3X6L55HSPNTKQ, seg_pos={0 0}, upload=455668]: Long tail cancellation due to optimal threshold Dec 27 06:51:38.967019748 (519h22m38.838102783s) [noise=true , address=45.91.116.8:4048 , piece_id=GP4RTD5A54JP6JTHHQRX7VCHMTEG2Z473MBIWDP26XIV4TFGELSA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 06:51:38.967028805 (519h22m38.83811185s) [noise=true , address=45.91.116.8:4048 , piece_id=GP4RTD5A54JP6JTHHQRX7VCHMTEG2Z473MBIWDP26XIV4TFGELSA, seg_pos={0 0}, upload=455668]: Long tail cancellation due to optimal threshold Dec 27 06:51:38.967057039 (519h22m38.838140084s) [noise=true , address=45.91.116.9:4261 , piece_id=LXNAYAZC2SEGM5RVU3UD755AEDJ3TFWJ4P2YCPM5C3UYEEHD2K7Q, seg_pos={0 0}, upload=455668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 06:51:38.967063380 (519h22m38.838146426s) [noise=true , address=45.91.116.9:4261 , piece_id=LXNAYAZC2SEGM5RVU3UD755AEDJ3TFWJ4P2YCPM5C3UYEEHD2K7Q, seg_pos={0 0}, upload=455668]: Long tail cancellation due to optimal threshold Dec 27 06:51:38.967110189 (519h22m38.838193224s) [seg_pos={0 0}, upload=455668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:51:38.967177007 (519h22m38.838260042s) [upload=455668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:51:38.967181635 (519h22m38.83826468s) [upload=455668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:51:39.252508324 (519h22m39.123591359s) [upload=455668]: Done uploading segments... Dec 27 06:51:39.252535035 (519h22m39.12361808s) [upload=455668]: Upload finished. err: <nil> Duration: 5.727419833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.........X...........XX.XXXXX......X.XX..XXX...X.XX........X] duration: 30.003910273s trace: https://jaeger.dev.storj.io/trace/28ecb31ed3cb3875 FAIL (2025-12-27 06:52:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: M5MPXLYGQTZTPLHMCKIYZOWUWG6BK7QMM2CL6QMYZQCGJWFAWSIA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: MXBE3O6WCZYOAQCTPNSJFL4UKWF564XOKHQRRVJGRVNC3S5BKFYA) context deadline exceeded ======LOGS====== Dec 27 06:51:35.701892242 (519h22m35.572975278s) [upload=455668]: Starting upload Dec 27 06:51:35.701954060 (519h22m35.573037095s) [upload=455668]: Done starting upload Dec 27 06:51:35.702123633 (519h22m35.573206678s) [upload=455668]: Uploading segments... Dec 27 06:51:35.702237990 (519h22m35.573321035s) [upload=455668]: Got next segment. Inline: false Dec 27 06:51:35.702279781 (519h22m35.573362816s) [upload=455668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:51:35.702285101 (519h22m35.573368136s) [upload=455668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:51:35.945197574 (519h22m35.816280609s) [seg_pos={0 0}, upload=455668]: Begin upload segment... Dec 27 06:51:35.945244723 (519h22m35.816327758s) [seg_pos={0 0}, upload=455668]: Done begin upload segment. Dec 27 06:51:35.945276845 (519h22m35.81635988s) [noise=true , address=145.79.227.13:4804 , piece_id=PEWVMEWQ2K3UH7LJQAHIVOT6LM5DLX6WMZTJC6T65AM3TZREC5OA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945307914 (519h22m35.816390949s) [noise=true , address=145.79.227.11:4731 , piece_id=YTCFQXWP5ERFXT4LTIXLNOEMGFGIO2Y5JQVOCENXM3D5NGLM7PWQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945645106 (519h22m35.816728141s) [noise=true , address=145.79.227.48:4302 , piece_id=B4YYGB7SQALIJLQB3M2AY6D7DWGKKVXGYO2DMW6YFW7KCYLKPELQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945655215 (519h22m35.81673825s) [noise=true , address=145.79.227.25:4617 , piece_id=RWWCSOURTNKK2SE5PLHSOKM4IPIY7EFJ2Q6IMEDMMOYV7R6QFFCQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945929858 (519h22m35.817012903s) [noise=true , address=45.91.116.8:4048 , piece_id=GP4RTD5A54JP6JTHHQRX7VCHMTEG2Z473MBIWDP26XIV4TFGELSA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.945949576 (519h22m35.817032611s) [noise=true , address=145.79.227.15:4874 , piece_id=FMGEYUJWDWODE5DE3HQ4JOK5REL4GFLEFCRUUYPT3N2FKEUVFNQQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946184673 (519h22m35.817267708s) [noise=true , address=145.79.227.37:4567 , piece_id=NVAZONYQXHP74WDR5LP2TTYE7RYMVSKNXHT4HS3IWCFRUUIRPZHA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946202297 (519h22m35.817285342s) [noise=true , address=77.90.31.17:4773 , piece_id=EANLWZ4BCHIE6VVHK2K2BOA7F2SOVSTKCTJOFRJ3X6L55HSPNTKQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946424901 (519h22m35.817507936s) [noise=true , address=145.79.227.29:4943 , piece_id=HNHZUPVW6GB5PSIJOYHJLMRWLK433GPOZTEGA2D35ONBQOZMJJIA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946451482 (519h22m35.817534517s) [noise=true , address=145.79.227.3:4151 , piece_id=TFBW5LBDUYN4FSD6CMHCOO32C4XT4NZ7UGHRDABOCCYWIOW4T3PQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946662954 (519h22m35.817745989s) [noise=true , address=77.90.31.11:4555 , piece_id=MXBE3O6WCZYOAQCTPNSJFL4UKWF564XOKHQRRVJGRVNC3S5BKFYA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946684155 (519h22m35.8177672s) [noise=true , address=145.79.227.46:4602 , piece_id=ABZ2HDEK5FZYWM6SGWMUFUJI2YQYMVENKUB3IEIG7SRTEPW3UEHQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946916657 (519h22m35.817999692s) [noise=true , address=145.79.227.2:4113 , piece_id=ZL4N6NAWNP4A2SZQQGVWV4G7U3RETFD2U275OFD4UCJSYDUTXZJQ, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.946951453 (519h22m35.818034498s) [noise=true , address=145.79.227.19:4079 , piece_id=KT6VYJUOAXAO6N35TF767DNL6WZKLYMQNBFWAE6I74QO3DJABUCA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.947187092 (519h22m35.818270127s) [noise=true , address=45.91.116.5:4300 , piece_id=M5MPXLYGQTZTPLHMCKIYZOWUWG6BK7QMM2CL6QMYZQCGJWFAWSIA, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:35.948002525 (519h22m35.81908556s) [noise=true , address=45.91.116.9:4261 , piece_id=LXNAYAZC2SEGM5RVU3UD755AEDJ3TFWJ4P2YCPM5C3UYEEHD2K7Q, seg_pos={0 0}, upload=455668]: Uploading piece... Dec 27 06:51:37.565621726 (519h22m37.436704771s) [upload=455668]: Next returned nil segment Dec 27 06:51:37.565635062 (519h22m37.436718097s) [upload=455668]: Waiting for error group managing segments... Dec 27 06:51:37.950016961 (519h22m37.821100006s) [noise=true , address=145.79.227.3:4151 , piece_id=TFBW5LBDUYN4FSD6CMHCOO32C4XT4NZ7UGHRDABOCCYWIOW4T3PQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.951331214 (519h22m37.822414259s) [noise=true , address=145.79.227.2:4113 , piece_id=ZL4N6NAWNP4A2SZQQGVWV4G7U3RETFD2U275OFD4UCJSYDUTXZJQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.952017250 (519h22m37.823100296s) [noise=true , address=145.79.227.37:4567 , piece_id=NVAZONYQXHP74WDR5LP2TTYE7RYMVSKNXHT4HS3IWCFRUUIRPZHA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.954793497 (519h22m37.825876542s) [noise=true , address=145.79.227.13:4804 , piece_id=PEWVMEWQ2K3UH7LJQAHIVOT6LM5DLX6WMZTJC6T65AM3TZREC5OA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.957724708 (519h22m37.828807754s) [noise=true , address=145.79.227.46:4602 , piece_id=ABZ2HDEK5FZYWM6SGWMUFUJI2YQYMVENKUB3IEIG7SRTEPW3UEHQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.957866007 (519h22m37.828949053s) [noise=true , address=145.79.227.11:4731 , piece_id=YTCFQXWP5ERFXT4LTIXLNOEMGFGIO2Y5JQVOCENXM3D5NGLM7PWQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.958337946 (519h22m37.829420981s) [noise=true , address=145.79.227.25:4617 , piece_id=RWWCSOURTNKK2SE5PLHSOKM4IPIY7EFJ2Q6IMEDMMOYV7R6QFFCQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.962947544 (519h22m37.834030589s) [noise=true , address=145.79.227.19:4079 , piece_id=KT6VYJUOAXAO6N35TF767DNL6WZKLYMQNBFWAE6I74QO3DJABUCA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.964975807 (519h22m37.836058842s) [noise=true , address=145.79.227.48:4302 , piece_id=B4YYGB7SQALIJLQB3M2AY6D7DWGKKVXGYO2DMW6YFW7KCYLKPELQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.968287202 (519h22m37.839370237s) [noise=true , address=145.79.227.15:4874 , piece_id=FMGEYUJWDWODE5DE3HQ4JOK5REL4GFLEFCRUUYPT3N2FKEUVFNQQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:37.976078558 (519h22m37.847161603s) [noise=true , address=145.79.227.29:4943 , piece_id=HNHZUPVW6GB5PSIJOYHJLMRWLK433GPOZTEGA2D35ONBQOZMJJIA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:38.948145758 (519h22m38.819228803s) [noise=true , address=77.90.31.11:4555 , piece_id=MXBE3O6WCZYOAQCTPNSJFL4UKWF564XOKHQRRVJGRVNC3S5BKFYA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:38.966636899 (519h22m38.837719934s) [noise=true , address=45.91.116.5:4300 , piece_id=M5MPXLYGQTZTPLHMCKIYZOWUWG6BK7QMM2CL6QMYZQCGJWFAWSIA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: <nil> Dec 27 06:51:38.966658028 (519h22m38.837741063s) [seg_pos={0 0}, upload=455668]: Segment reached optimal threshold of 13 pieces. Dec 27 06:51:38.966866135 (519h22m38.83794917s) [noise=true , address=77.90.31.17:4773 , piece_id=EANLWZ4BCHIE6VVHK2K2BOA7F2SOVSTKCTJOFRJ3X6L55HSPNTKQ, seg_pos={0 0}, upload=455668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 27 06:51:38.966874371 (519h22m38.837957406s) [noise=true , address=77.90.31.17:4773 , piece_id=EANLWZ4BCHIE6VVHK2K2BOA7F2SOVSTKCTJOFRJ3X6L55HSPNTKQ, seg_pos={0 0}, upload=455668]: Long tail cancellation due to optimal threshold Dec 27 06:51:38.967019748 (519h22m38.838102783s) [noise=true , address=45.91.116.8:4048 , piece_id=GP4RTD5A54JP6JTHHQRX7VCHMTEG2Z473MBIWDP26XIV4TFGELSA, seg_pos={0 0}, upload=455668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 06:51:38.967028805 (519h22m38.83811185s) [noise=true , address=45.91.116.8:4048 , piece_id=GP4RTD5A54JP6JTHHQRX7VCHMTEG2Z473MBIWDP26XIV4TFGELSA, seg_pos={0 0}, upload=455668]: Long tail cancellation due to optimal threshold Dec 27 06:51:38.967057039 (519h22m38.838140084s) [noise=true , address=45.91.116.9:4261 , piece_id=LXNAYAZC2SEGM5RVU3UD755AEDJ3TFWJ4P2YCPM5C3UYEEHD2K7Q, seg_pos={0 0}, upload=455668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 27 06:51:38.967063380 (519h22m38.838146426s) [noise=true , address=45.91.116.9:4261 , piece_id=LXNAYAZC2SEGM5RVU3UD755AEDJ3TFWJ4P2YCPM5C3UYEEHD2K7Q, seg_pos={0 0}, upload=455668]: Long tail cancellation due to optimal threshold Dec 27 06:51:38.967110189 (519h22m38.838193224s) [seg_pos={0 0}, upload=455668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:51:38.967177007 (519h22m38.838260042s) [upload=455668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:51:38.967181635 (519h22m38.83826468s) [upload=455668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:51:39.252508324 (519h22m39.123591359s) [upload=455668]: Done uploading segments... Dec 27 06:51:39.252535035 (519h22m39.12361808s) [upload=455668]: Upload finished. err: <nil> Duration: 30.00395603s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 51.192572803s trace: https://jaeger.dev.storj.io/trace/7637d58679dfc2b FAIL (2025-12-27 06:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, Piece ID: 4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ) context deadline exceeded; piecestore: (Node ID: 1XuRKaSZC9qb9M17ctkcNkf5dSkBYpDBHgxMS56wThuw8s2R6, Piece ID: M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ) context deadline exceeded; piecestore: (Node ID: 12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o, Piece ID: J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A) context deadline exceeded ======LOGS====== ec 27 06:23:01.300606186 (518h54m1.171689221s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300678243 (518h54m1.171761288s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300726415 (518h54m1.17180946s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300775528 (518h54m1.171858573s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300817849 (518h54m1.171900884s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300861301 (518h54m1.171944346s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.300904174 (518h54m1.171987219s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.302497919 (518h54m1.173580954s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303015062 (518h54m1.174098117s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.303683576 (518h54m1.174766621s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304243622 (518h54m1.175326668s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.304804771 (518h54m1.175887816s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.305452815 (518h54m1.17653586s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.307203358 (518h54m1.178286393s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.308885732 (518h54m1.179968767s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:01.310641255 (518h54m1.1817243s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Uploading piece... Dec 27 06:23:02.661676879 (518h54m2.532759924s) [upload=455252]: Next returned nil segment Dec 27 06:23:02.661697388 (518h54m2.532780434s) [upload=455252]: Waiting for error group managing segments... Dec 27 06:23:03.006040636 (518h54m2.877123681s) [noise=true , address=64.49.13.155:4918 , piece_id=HWHYUPUIYB6LV6A67GCTB6QLF3EISWHVZ6VJPTMNNPLTERYJLVGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.198726907 (518h54m3.069809952s) [noise=true , address=145.79.227.46:4623 , piece_id=TPXBBM24PW2TBTKLHOL3GHGNY6AD3KY6ICNVNF3U2BKBBFCGPCVA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.209535790 (518h54m3.080618825s) [noise=true , address=145.79.227.29:4953 , piece_id=WTFPI4JRATOX4HGQO3NMPCKUS5JCVOQXUHMTUOIC4GOLJ2RBU7YA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.396894026 (518h54m3.267977071s) [noise=true , address=77.90.31.10:4549 , piece_id=QO5DB3KALGHWLUCMNHRWWDYPJDX7RHYATNQJINUIJNZUEWF67Q5A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.444090588 (518h54m3.315173633s) [noise=true , address=77.90.31.18:4840 , piece_id=6JMN5TN73IUHYLMDJSKMJP5R3RINYY3PB7WG37FCGLF6MVBLVOYQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.453033817 (518h54m3.324116862s) [noise=true , address=77.90.31.16:4762 , piece_id=4XCNHC4MRKTSNZC34RD4WQ5DV63MKTDHTHF7OYMR7WMB627O6TSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.461352988 (518h54m3.332436033s) [noise=true , address=77.90.31.12:4618 , piece_id=M2TENJN3VFPE6KIAMVMXZZM2FDMDEZQT7E4UGQ7YUHYGRC5ENBGQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:03.601936156 (518h54m3.473019201s) [noise=true , address=77.90.31.14:4681 , piece_id=ERXJPMISQ4RR77EUBAGJ5X6J754BAERZMJ5HMSEY2EWLFSQEOE6A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.076565583 (518h54m3.947648658s) [noise=true , address=77.90.31.15:4716 , piece_id=RHWQ6MFKLA2SHVHZGPUQJU77P2PE3YU4PZ3ADBLRVOJ7K37GANSQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.116623061 (518h54m3.987706106s) [noise=true , address=145.79.227.40:4067 , piece_id=KCDN2L4CIRQAJKQSL36NHVI55PZEH6FSS2ZW6ITUTHSAENO3CLIQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.121726017 (518h54m3.992809062s) [noise=true , address=77.90.31.13:4651 , piece_id=J4ASNWNAVCOFX34R2FFAZ2M7H5E2IAIOCYWJHWULXDE53TGZUX2A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.132649729 (518h54m4.003732784s) [noise=true , address=145.79.227.44:4208 , piece_id=KCUYP5TL52QP7MAUU2HIUBIQD6ZWTT2G2GAC2RT4LFR7ECTEWWMA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.139015891 (518h54m4.010098936s) [noise=true , address=145.79.227.43:4172 , piece_id=UYDTZF562756XZFBJO3FKVWZ57HBZ7W33OUOUJHECY4HKY3CCWZA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169389909 (518h54m4.040472954s) [noise=true , address=145.79.227.30:4795 , piece_id=M3JOR4AR5FEN7VGIN3U76OBFT2Y3FLFJWUDIDJELU7UG3O4EF2FQ, seg_pos={0 0}, upload=455252]: Done uploading piece. err: <nil> Dec 27 06:23:04.169423092 (518h54m4.040506137s) [seg_pos={0 0}, upload=455252]: Segment reached optimal threshold of 14 pieces. Dec 27 06:23:04.169650115 (518h54m4.04073316s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47858->64.49.13.169:4706: use of closed network connection Dec 27 06:23:04.169661276 (518h54m4.040744321s) [noise=true , address=64.49.13.169:4706 , piece_id=H32KZR6DBDBJ357IQKMUFJ6WUEXKIYHAGUCLK3FVA5XXULT33Z3Q, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169734896 (518h54m4.040817961s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169756528 (518h54m4.040839583s) [noise=true , address=145.79.227.33:4890 , piece_id=L2SKS7NYGWMNNHQ23RAHGVLQVT6LVNIYDZITZV3H37NQRDR2EQTA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.169892065 (518h54m4.04097512s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126JXfJvuad1JedXjgEJuA12awNtwHBYNZhHSJoei2ZYTrccDmA): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.169902125 (518h54m4.04098518s) [noise=true , address=45.91.116.6:4990 , piece_id=MIF6WEJOUHNOPVUFFA47E34RP325JABK2CRWZQ6EATQM4GXQAZRA, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170085444 (518h54m4.041168489s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 27 06:23:04.170094631 (518h54m4.041177676s) [noise=true , address=145.79.227.34:4918 , piece_id=YPRARW3ZBT5NMHYDU6DFZIGX42H23ISZXANZLHYGATMQ6Y4XPE7A, seg_pos={0 0}, upload=455252]: Long tail cancellation due to optimal threshold Dec 27 06:23:04.170205322 (518h54m4.041288367s) [seg_pos={0 0}, upload=455252]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:23:04.170345309 (518h54m4.041428364s) [upload=455252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:23:04.170352803 (518h54m4.041435848s) [upload=455252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:23:04.479479019 (518h54m4.350562065s) [upload=455252]: Done uploading segments... Dec 27 06:23:04.479517072 (518h54m4.350600107s) [upload=455252]: Upload finished. err: <nil> Duration: 51.192593704s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)