Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........X.X.XX.XX..XX.........X..X......X..................] duration: 27.929661729s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 27.929689602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..........X.X.XX.XX..XX.........X..X......X.................] duration: 15.671534907s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 15.671558832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 15.503455506s trace: https://jaeger.dev.storj.io/trace/437161d9c2485949 FAIL (2025-12-30 23:45:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=Q4ELTYLCBR7R2JKDYS726H3FETOKCMU2XTDON2YOPZYCXCK4SCZA, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.956596797 (608h15m17.827679832s) [noise=true , address=45.91.116.8:4076 , piece_id=E5JTEUN5GJIZK2ZR5JHIWYGXZ4I3NSM3ZPD5YM4WAPNIXQJY736Q, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.956678693 (608h15m17.827761728s) [noise=true , address=145.79.227.3:4180 , piece_id=3NYJ4EQ3ZNRPP2HU4FJPQZH356QTQCB67KLUOU4LYTAIAON4LLMQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.956979205 (608h15m17.82806224s) [noise=true , address=45.91.116.11:4122 , piece_id=Q7DPCERARVTCHZBSU4RHOFYFQUNCLWWWKD7IGPKXJ7JVL3S3UN4A, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.957204013 (608h15m17.828287058s) [noise=true , address=145.79.227.30:4796 , piece_id=7RLER6D3SSYGZ4JMBVR7JYVSFZQLEGBFWGGHADAILIN44TYPHB7A, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.957331626 (608h15m17.828414661s) [noise=true , address=64.49.13.27:4965 , piece_id=4WDSAXHIMHWXE7PI4HC7KV7SXND7DBCLPSGU46D4ICMKM2NDGKXQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.957775661 (608h15m17.828858696s) [noise=true , address=145.79.227.21:4484 , piece_id=GLYNILFEX7232BECJASCPZHIU6GGXPTDU4BHXR3UEWMRTLZTZHGA, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.958406392 (608h15m17.829489437s) [noise=true , address=77.90.31.13:4644 , piece_id=X7JUQL5GSHWGNC3E33X5HGRDAXXXRRIBRXSRAVLF6ED4RUXDZ4YQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.962322639 (608h15m17.833405684s) [noise=true , address=145.79.227.40:4072 , piece_id=ZYLA4DIIZYWG3ZV5M2HBEH2X3QOQECYEKZSPE7BY43OQCW2V2VBQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.963728406 (608h15m17.834811441s) [noise=true , address=77.90.31.17:4769 , piece_id=UF5JMXHL5WC4HKBQJH3Y52KS2IWTSSUG7ZQ2IIBEB36PCJ7VYKJQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.964967114 (608h15m17.836050149s) [noise=true , address=64.49.13.169:4707 , piece_id=KQXN6UXTKEVKW6PLWAMBT4MDQJNI6RDHI7TRUACBIM7JOIYX5D2A, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.967108192 (608h15m17.838191227s) [noise=true , address=5.231.73.3:4022 , piece_id=MBUHE2KSIVSNODWQBXZ7HII5Z6GK5GUDVGYBP2FNAFH3EA62DNHQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.975679341 (608h15m17.846762376s) [noise=true , address=193.149.164.143:4700 , piece_id=X6SW4NV6IPV63M7LRDLZJCJWCQM5QXT3DLL4DQC5UUNDTC3LS2ZQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.977609076 (608h15m17.848692111s) [noise=true , address=45.91.116.7:4031 , piece_id=6ADEQ7TB7MRQWHAPPAQT5LHZU3MRTTF5DMBNZX3EGPZ5E5KYL7VA, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.979415235 (608h15m17.85049827s) [noise=true , address=145.79.227.11:4750 , piece_id=KXYGLKEOL7B6LXGQCA7NFVJIG7H5UDSBFGE5YBM6EMCWCDMDIEZQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:19.324096096 (608h15m19.195179131s) [upload=533074]: Next returned nil segment Dec 30 23:44:19.324108249 (608h15m19.195191284s) [upload=533074]: Waiting for error group managing segments... Dec 30 23:44:19.692038203 (608h15m19.563121248s) [noise=true , address=64.49.13.27:4965 , piece_id=4WDSAXHIMHWXE7PI4HC7KV7SXND7DBCLPSGU46D4ICMKM2NDGKXQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.798665391 (608h15m19.669748426s) [noise=true , address=145.79.227.3:4180 , piece_id=3NYJ4EQ3ZNRPP2HU4FJPQZH356QTQCB67KLUOU4LYTAIAON4LLMQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.833390000 (608h15m19.704473035s) [noise=true , address=145.79.227.11:4750 , piece_id=KXYGLKEOL7B6LXGQCA7NFVJIG7H5UDSBFGE5YBM6EMCWCDMDIEZQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.917047451 (608h15m19.788130486s) [noise=true , address=5.231.73.3:4022 , piece_id=MBUHE2KSIVSNODWQBXZ7HII5Z6GK5GUDVGYBP2FNAFH3EA62DNHQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.991484928 (608h15m19.862567963s) [noise=true , address=77.90.31.17:4769 , piece_id=UF5JMXHL5WC4HKBQJH3Y52KS2IWTSSUG7ZQ2IIBEB36PCJ7VYKJQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.083878504 (608h15m19.954961549s) [noise=true , address=77.90.31.13:4644 , piece_id=X7JUQL5GSHWGNC3E33X5HGRDAXXXRRIBRXSRAVLF6ED4RUXDZ4YQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.087305018 (608h15m19.958388053s) [noise=true , address=77.90.31.18:4839 , piece_id=FZQBSKRHEO3QFSFEBOUDVFB3UWVTWGJGWV5ESJDIRRHROAYGF3ZQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.127405584 (608h15m19.998488619s) [noise=true , address=45.91.116.8:4076 , piece_id=E5JTEUN5GJIZK2ZR5JHIWYGXZ4I3NSM3ZPD5YM4WAPNIXQJY736Q, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.128917041 (608h15m20.000000076s) [noise=true , address=45.91.116.7:4031 , piece_id=6ADEQ7TB7MRQWHAPPAQT5LHZU3MRTTF5DMBNZX3EGPZ5E5KYL7VA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.197235697 (608h15m20.068318742s) [noise=true , address=45.91.116.11:4122 , piece_id=Q7DPCERARVTCHZBSU4RHOFYFQUNCLWWWKD7IGPKXJ7JVL3S3UN4A, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.688206322 (608h15m20.559289367s) [noise=true , address=145.79.227.30:4796 , piece_id=7RLER6D3SSYGZ4JMBVR7JYVSFZQLEGBFWGGHADAILIN44TYPHB7A, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.691551281 (608h15m20.562634317s) [noise=true , address=145.79.227.29:4958 , piece_id=NU5UW4EOPE2NIO7RY72OY3KCOKDNLPMVAQWXAHXH3GAYUKZ2JCGQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.721448849 (608h15m20.592531884s) [noise=true , address=145.79.227.49:4367 , piece_id=Q4ELTYLCBR7R2JKDYS726H3FETOKCMU2XTDON2YOPZYCXCK4SCZA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.731964031 (608h15m20.603047066s) [noise=true , address=193.149.164.143:4700 , piece_id=X6SW4NV6IPV63M7LRDLZJCJWCQM5QXT3DLL4DQC5UUNDTC3LS2ZQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.731983298 (608h15m20.603066343s) [seg_pos={0 0}, upload=533074]: Segment reached optimal threshold of 14 pieces. Dec 30 23:44:20.732169271 (608h15m20.603252306s) [noise=true , address=145.79.227.21:4484 , piece_id=GLYNILFEX7232BECJASCPZHIU6GGXPTDU4BHXR3UEWMRTLZTZHGA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN): protocol: write tcp 188.166.225.170:32934->145.79.227.21:4484: use of closed network connection; write tcp 188.166.225.170:32934->145.79.227.21:4484: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32934->145.79.227.21:4484: use of closed network connection Dec 30 23:44:20.732175173 (608h15m20.603258208s) [noise=true , address=145.79.227.21:4484 , piece_id=GLYNILFEX7232BECJASCPZHIU6GGXPTDU4BHXR3UEWMRTLZTZHGA, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732307555 (608h15m20.60339059s) [noise=true , address=145.79.227.40:4072 , piece_id=ZYLA4DIIZYWG3ZV5M2HBEH2X3QOQECYEKZSPE7BY43OQCW2V2VBQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 23:44:20.732324157 (608h15m20.603407192s) [noise=true , address=145.79.227.40:4072 , piece_id=ZYLA4DIIZYWG3ZV5M2HBEH2X3QOQECYEKZSPE7BY43OQCW2V2VBQ, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732344074 (608h15m20.603427109s) [noise=true , address=64.49.13.169:4707 , piece_id=KQXN6UXTKEVKW6PLWAMBT4MDQJNI6RDHI7TRUACBIM7JOIYX5D2A, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49228->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49228->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->64.49.13.169:4707: use of closed network connection Dec 30 23:44:20.732349966 (608h15m20.603433001s) [noise=true , address=64.49.13.169:4707 , piece_id=KQXN6UXTKEVKW6PLWAMBT4MDQJNI6RDHI7TRUACBIM7JOIYX5D2A, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732511042 (608h15m20.603594077s) [noise=true , address=145.79.227.7:4301 , piece_id=ODAIXQ2ER5BKSIAW6YEAFZLO7MPKFQGCTRO2SZUQB7MWNNKL6BWA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG): protocol: expected piece hash; context canceled; EOF Dec 30 23:44:20.732518146 (608h15m20.603601181s) [noise=true , address=145.79.227.7:4301 , piece_id=ODAIXQ2ER5BKSIAW6YEAFZLO7MPKFQGCTRO2SZUQB7MWNNKL6BWA, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732535409 (608h15m20.603618444s) [seg_pos={0 0}, upload=533074]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 23:44:20.732625620 (608h15m20.603708655s) [upload=533074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:44:20.732630631 (608h15m20.603713666s) [upload=533074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:44:21.024075710 (608h15m20.895158765s) [upload=533074]: Done uploading segments... Dec 30 23:44:21.024101429 (608h15m20.895184464s) [upload=533074]: Upload finished. err: <nil> Duration: 15.503470315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.005121589s trace: https://jaeger.dev.storj.io/trace/437161d9c2485949 FAIL (2025-12-30 23:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=Q4ELTYLCBR7R2JKDYS726H3FETOKCMU2XTDON2YOPZYCXCK4SCZA, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.956596797 (608h15m17.827679832s) [noise=true , address=45.91.116.8:4076 , piece_id=E5JTEUN5GJIZK2ZR5JHIWYGXZ4I3NSM3ZPD5YM4WAPNIXQJY736Q, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.956678693 (608h15m17.827761728s) [noise=true , address=145.79.227.3:4180 , piece_id=3NYJ4EQ3ZNRPP2HU4FJPQZH356QTQCB67KLUOU4LYTAIAON4LLMQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.956979205 (608h15m17.82806224s) [noise=true , address=45.91.116.11:4122 , piece_id=Q7DPCERARVTCHZBSU4RHOFYFQUNCLWWWKD7IGPKXJ7JVL3S3UN4A, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.957204013 (608h15m17.828287058s) [noise=true , address=145.79.227.30:4796 , piece_id=7RLER6D3SSYGZ4JMBVR7JYVSFZQLEGBFWGGHADAILIN44TYPHB7A, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.957331626 (608h15m17.828414661s) [noise=true , address=64.49.13.27:4965 , piece_id=4WDSAXHIMHWXE7PI4HC7KV7SXND7DBCLPSGU46D4ICMKM2NDGKXQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.957775661 (608h15m17.828858696s) [noise=true , address=145.79.227.21:4484 , piece_id=GLYNILFEX7232BECJASCPZHIU6GGXPTDU4BHXR3UEWMRTLZTZHGA, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.958406392 (608h15m17.829489437s) [noise=true , address=77.90.31.13:4644 , piece_id=X7JUQL5GSHWGNC3E33X5HGRDAXXXRRIBRXSRAVLF6ED4RUXDZ4YQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.962322639 (608h15m17.833405684s) [noise=true , address=145.79.227.40:4072 , piece_id=ZYLA4DIIZYWG3ZV5M2HBEH2X3QOQECYEKZSPE7BY43OQCW2V2VBQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.963728406 (608h15m17.834811441s) [noise=true , address=77.90.31.17:4769 , piece_id=UF5JMXHL5WC4HKBQJH3Y52KS2IWTSSUG7ZQ2IIBEB36PCJ7VYKJQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.964967114 (608h15m17.836050149s) [noise=true , address=64.49.13.169:4707 , piece_id=KQXN6UXTKEVKW6PLWAMBT4MDQJNI6RDHI7TRUACBIM7JOIYX5D2A, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.967108192 (608h15m17.838191227s) [noise=true , address=5.231.73.3:4022 , piece_id=MBUHE2KSIVSNODWQBXZ7HII5Z6GK5GUDVGYBP2FNAFH3EA62DNHQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.975679341 (608h15m17.846762376s) [noise=true , address=193.149.164.143:4700 , piece_id=X6SW4NV6IPV63M7LRDLZJCJWCQM5QXT3DLL4DQC5UUNDTC3LS2ZQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.977609076 (608h15m17.848692111s) [noise=true , address=45.91.116.7:4031 , piece_id=6ADEQ7TB7MRQWHAPPAQT5LHZU3MRTTF5DMBNZX3EGPZ5E5KYL7VA, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:17.979415235 (608h15m17.85049827s) [noise=true , address=145.79.227.11:4750 , piece_id=KXYGLKEOL7B6LXGQCA7NFVJIG7H5UDSBFGE5YBM6EMCWCDMDIEZQ, seg_pos={0 0}, upload=533074]: Uploading piece... Dec 30 23:44:19.324096096 (608h15m19.195179131s) [upload=533074]: Next returned nil segment Dec 30 23:44:19.324108249 (608h15m19.195191284s) [upload=533074]: Waiting for error group managing segments... Dec 30 23:44:19.692038203 (608h15m19.563121248s) [noise=true , address=64.49.13.27:4965 , piece_id=4WDSAXHIMHWXE7PI4HC7KV7SXND7DBCLPSGU46D4ICMKM2NDGKXQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.798665391 (608h15m19.669748426s) [noise=true , address=145.79.227.3:4180 , piece_id=3NYJ4EQ3ZNRPP2HU4FJPQZH356QTQCB67KLUOU4LYTAIAON4LLMQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.833390000 (608h15m19.704473035s) [noise=true , address=145.79.227.11:4750 , piece_id=KXYGLKEOL7B6LXGQCA7NFVJIG7H5UDSBFGE5YBM6EMCWCDMDIEZQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.917047451 (608h15m19.788130486s) [noise=true , address=5.231.73.3:4022 , piece_id=MBUHE2KSIVSNODWQBXZ7HII5Z6GK5GUDVGYBP2FNAFH3EA62DNHQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:19.991484928 (608h15m19.862567963s) [noise=true , address=77.90.31.17:4769 , piece_id=UF5JMXHL5WC4HKBQJH3Y52KS2IWTSSUG7ZQ2IIBEB36PCJ7VYKJQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.083878504 (608h15m19.954961549s) [noise=true , address=77.90.31.13:4644 , piece_id=X7JUQL5GSHWGNC3E33X5HGRDAXXXRRIBRXSRAVLF6ED4RUXDZ4YQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.087305018 (608h15m19.958388053s) [noise=true , address=77.90.31.18:4839 , piece_id=FZQBSKRHEO3QFSFEBOUDVFB3UWVTWGJGWV5ESJDIRRHROAYGF3ZQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.127405584 (608h15m19.998488619s) [noise=true , address=45.91.116.8:4076 , piece_id=E5JTEUN5GJIZK2ZR5JHIWYGXZ4I3NSM3ZPD5YM4WAPNIXQJY736Q, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.128917041 (608h15m20.000000076s) [noise=true , address=45.91.116.7:4031 , piece_id=6ADEQ7TB7MRQWHAPPAQT5LHZU3MRTTF5DMBNZX3EGPZ5E5KYL7VA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.197235697 (608h15m20.068318742s) [noise=true , address=45.91.116.11:4122 , piece_id=Q7DPCERARVTCHZBSU4RHOFYFQUNCLWWWKD7IGPKXJ7JVL3S3UN4A, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.688206322 (608h15m20.559289367s) [noise=true , address=145.79.227.30:4796 , piece_id=7RLER6D3SSYGZ4JMBVR7JYVSFZQLEGBFWGGHADAILIN44TYPHB7A, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.691551281 (608h15m20.562634317s) [noise=true , address=145.79.227.29:4958 , piece_id=NU5UW4EOPE2NIO7RY72OY3KCOKDNLPMVAQWXAHXH3GAYUKZ2JCGQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.721448849 (608h15m20.592531884s) [noise=true , address=145.79.227.49:4367 , piece_id=Q4ELTYLCBR7R2JKDYS726H3FETOKCMU2XTDON2YOPZYCXCK4SCZA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.731964031 (608h15m20.603047066s) [noise=true , address=193.149.164.143:4700 , piece_id=X6SW4NV6IPV63M7LRDLZJCJWCQM5QXT3DLL4DQC5UUNDTC3LS2ZQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: <nil> Dec 30 23:44:20.731983298 (608h15m20.603066343s) [seg_pos={0 0}, upload=533074]: Segment reached optimal threshold of 14 pieces. Dec 30 23:44:20.732169271 (608h15m20.603252306s) [noise=true , address=145.79.227.21:4484 , piece_id=GLYNILFEX7232BECJASCPZHIU6GGXPTDU4BHXR3UEWMRTLZTZHGA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN): protocol: write tcp 188.166.225.170:32934->145.79.227.21:4484: use of closed network connection; write tcp 188.166.225.170:32934->145.79.227.21:4484: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32934->145.79.227.21:4484: use of closed network connection Dec 30 23:44:20.732175173 (608h15m20.603258208s) [noise=true , address=145.79.227.21:4484 , piece_id=GLYNILFEX7232BECJASCPZHIU6GGXPTDU4BHXR3UEWMRTLZTZHGA, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732307555 (608h15m20.60339059s) [noise=true , address=145.79.227.40:4072 , piece_id=ZYLA4DIIZYWG3ZV5M2HBEH2X3QOQECYEKZSPE7BY43OQCW2V2VBQ, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne): protocol: expected piece hash; context canceled; EOF Dec 30 23:44:20.732324157 (608h15m20.603407192s) [noise=true , address=145.79.227.40:4072 , piece_id=ZYLA4DIIZYWG3ZV5M2HBEH2X3QOQECYEKZSPE7BY43OQCW2V2VBQ, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732344074 (608h15m20.603427109s) [noise=true , address=64.49.13.169:4707 , piece_id=KQXN6UXTKEVKW6PLWAMBT4MDQJNI6RDHI7TRUACBIM7JOIYX5D2A, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49228->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49228->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->64.49.13.169:4707: use of closed network connection Dec 30 23:44:20.732349966 (608h15m20.603433001s) [noise=true , address=64.49.13.169:4707 , piece_id=KQXN6UXTKEVKW6PLWAMBT4MDQJNI6RDHI7TRUACBIM7JOIYX5D2A, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732511042 (608h15m20.603594077s) [noise=true , address=145.79.227.7:4301 , piece_id=ODAIXQ2ER5BKSIAW6YEAFZLO7MPKFQGCTRO2SZUQB7MWNNKL6BWA, seg_pos={0 0}, upload=533074]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG): protocol: expected piece hash; context canceled; EOF Dec 30 23:44:20.732518146 (608h15m20.603601181s) [noise=true , address=145.79.227.7:4301 , piece_id=ODAIXQ2ER5BKSIAW6YEAFZLO7MPKFQGCTRO2SZUQB7MWNNKL6BWA, seg_pos={0 0}, upload=533074]: Long tail cancellation due to optimal threshold Dec 30 23:44:20.732535409 (608h15m20.603618444s) [seg_pos={0 0}, upload=533074]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 23:44:20.732625620 (608h15m20.603708655s) [upload=533074]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:44:20.732630631 (608h15m20.603713666s) [upload=533074]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:44:21.024075710 (608h15m20.895158765s) [upload=533074]: Done uploading segments... Dec 30 23:44:21.024101429 (608h15m20.895184464s) [upload=533074]: Upload finished. err: <nil> Duration: 1m0.005141497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...........X.X.XX.XX..XX.........X..X......X................] duration: 6.112057596s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:44:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.112083736s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [............X.X.XX.XX..XX.........X..X......X...............] duration: 6.047843725s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:44:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.047873762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.............X.X.XX.XX..XX.........X..X......X..............] duration: 6.77527361s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:43:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.775301734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..............X.X.XX.XX..XX.........X..X......X.............] duration: 5.88577506s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:43:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.885791372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X.X.XX.XX..XX.........X..X......X............] duration: 5.985284667s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.985301718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................X.X.XX.XX..XX.........X..X......X...........] duration: 16.072457763s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 16.072487261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................X.X.XX.XX..XX.........X..X......X..........] duration: 5.883338427s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:40:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.883383993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................X.X.XX.XX..XX.........X..X......X.........] duration: 13.733484289s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:40:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 13.733526711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 31.875588608s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:40:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 31.875618374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 15.380793092s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.380821968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...................X.X.XX.XX..XX.........X..X......X........] duration: 6.138856336s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.138882646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 38.837139679s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:39:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 38.837154868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 29.322263381s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 29.322291165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....................X.X.XX.XX..XX.........X..X......X.......] duration: 6.654261158s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.654303901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 22.442474164s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:37:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 22.442500884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 23.506219681s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 23.506244928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....................X.X.XX.XX..XX.........X..X......X......] duration: 6.26271003s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:37:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.262740399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....................X.X.XX.XX..XX.........X..X......X.....] duration: 6.657799138s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.657829066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....................X.X.XX.XX..XX.........X..X......X....] duration: 5.977189669s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:36:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.977211322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.253536264s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.253561892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.....................X.X.XX.XX..XX.........X..X......X...] duration: 11.850903285s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 11.85092211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 20.83510189s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 20.835118801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 29.838482558s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:35:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 29.838510562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X.....................X.X.XX.XX..XX.........X..X......X..] duration: 7.67187508s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:34:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 7.671900107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 16.119217861s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 16.119241997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X.....................X.X.XX.XX..XX.........X..X......X.] duration: 5.930689288s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:33:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.930713205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X.....................X.X.XX.XX..XX.........X..X......X] duration: 30.003301347s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:33:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 30.00332417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 43.863959513s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:33:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 43.863981594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 48.280758753s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:32:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 48.280786384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.X.....................X.X.XX.XX..XX.........X..X......] duration: 6.865869969s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:32:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 6.865895947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X.....................X.X.XX.XX..XX.........X..X.....] duration: 15.889344776s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 15.889376545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 19.315281081s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 19.315310968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.X.....................X.X.XX.XX..XX.........X..X....] duration: 7.106070397s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 7.106086679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.X.....................X.X.XX.XX..XX.........X..X...] duration: 28.928554418s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:30:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 28.928579967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.31524994s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:30:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.315267684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 25.145206296s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:29:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 25.145222647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X.X.....................X.X.XX.XX..XX.........X..X..] duration: 5.927783616s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:29:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 5.927810556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 31.633212026s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:29:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 31.633237845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X........X.X.....................X.X.XX.XX..XX.........X..X.] duration: 6.029026756s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:28:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 6.029073685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 36.198460932s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 36.198496299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X........X.X.....................X.X.XX.XX..XX.........X..X] duration: 30.00252539s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:27:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 30.002547963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 18.353761448s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:27:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 18.35379405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 23.730632257s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:26:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 23.730662856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X........X.X.....................X.X.XX.XX..XX.........X..] duration: 9.039778361s trace: https://jaeger.dev.storj.io/trace/5df0eedd8f155eba FAIL (2025-12-30 23:26:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ) context deadline exceeded ======LOGS====== d=532790]: Starting upload Dec 30 23:25:01.585730368 (607h56m1.456813413s) [upload=532790]: Done starting upload Dec 30 23:25:01.585844787 (607h56m1.456927832s) [upload=532790]: Uploading segments... Dec 30 23:25:01.586055338 (607h56m1.457138383s) [upload=532790]: Got next segment. Inline: false Dec 30 23:25:01.586138035 (607h56m1.45722108s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:25:01.586148915 (607h56m1.45723197s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:25:01.836805332 (607h56m1.707888387s) [seg_pos={0 0}, upload=532790]: Begin upload segment... Dec 30 23:25:01.836872830 (607h56m1.707955875s) [seg_pos={0 0}, upload=532790]: Done begin upload segment. Dec 30 23:25:01.836915182 (607h56m1.707998227s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839532234 (607h56m1.71061529s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839649688 (607h56m1.710732743s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839713740 (607h56m1.710796785s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840244561 (607h56m1.711327626s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840611871 (607h56m1.711694916s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840726599 (607h56m1.711809644s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841171677 (607h56m1.712254722s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841199178 (607h56m1.712282223s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841668743 (607h56m1.712751788s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841696295 (607h56m1.712779341s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842184876 (607h56m1.713267921s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842429782 (607h56m1.713512817s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843206401 (607h56m1.714289446s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843831721 (607h56m1.714914766s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.844407488 (607h56m1.715490523s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:03.540346597 (607h56m3.411429653s) [upload=532790]: Next returned nil segment Dec 30 23:25:03.540374801 (607h56m3.411457866s) [upload=532790]: Waiting for error group managing segments... Dec 30 23:25:03.824335882 (607h56m3.695418927s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.832981394 (607h56m3.704064459s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.849417580 (607h56m3.720500625s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.866091639 (607h56m3.737174684s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.893000355 (607h56m3.764083401s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.929223007 (607h56m3.800306052s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.930027409 (607h56m3.801110454s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.026266778 (607h56m3.897349824s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.142674381 (607h56m4.013757426s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.156619473 (607h56m4.027702518s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.177527193 (607h56m4.048610238s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.249650634 (607h56m4.120733679s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320523523 (607h56m4.191606568s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320604697 (607h56m4.191687752s) [seg_pos={0 0}, upload=532790]: Segment reached optimal threshold of 13 pieces. Dec 30 23:25:04.321090162 (607h56m4.192173237s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:55884->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 23:25:04.321102807 (607h56m4.192185852s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321187618 (607h56m4.192270663s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection Dec 30 23:25:04.321199440 (607h56m4.192282485s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321436391 (607h56m4.192519446s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 23:25:04.321446772 (607h56m4.192529817s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321473181 (607h56m4.192556226s) [seg_pos={0 0}, upload=532790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:25:04.321595024 (607h56m4.192678069s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:25:04.321604051 (607h56m4.192687096s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:25:04.614658617 (607h56m4.485741662s) [upload=532790]: Done uploading segments... Dec 30 23:25:04.614685839 (607h56m4.485768884s) [upload=532790]: Upload finished. err: <nil> Duration: 9.039807655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X........X.X.....................X.X.XX.XX..XX.........X.] duration: 5.864190944s trace: https://jaeger.dev.storj.io/trace/5df0eedd8f155eba FAIL (2025-12-30 23:25:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ) context deadline exceeded ======LOGS====== d=532790]: Starting upload Dec 30 23:25:01.585730368 (607h56m1.456813413s) [upload=532790]: Done starting upload Dec 30 23:25:01.585844787 (607h56m1.456927832s) [upload=532790]: Uploading segments... Dec 30 23:25:01.586055338 (607h56m1.457138383s) [upload=532790]: Got next segment. Inline: false Dec 30 23:25:01.586138035 (607h56m1.45722108s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:25:01.586148915 (607h56m1.45723197s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:25:01.836805332 (607h56m1.707888387s) [seg_pos={0 0}, upload=532790]: Begin upload segment... Dec 30 23:25:01.836872830 (607h56m1.707955875s) [seg_pos={0 0}, upload=532790]: Done begin upload segment. Dec 30 23:25:01.836915182 (607h56m1.707998227s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839532234 (607h56m1.71061529s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839649688 (607h56m1.710732743s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839713740 (607h56m1.710796785s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840244561 (607h56m1.711327626s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840611871 (607h56m1.711694916s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840726599 (607h56m1.711809644s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841171677 (607h56m1.712254722s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841199178 (607h56m1.712282223s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841668743 (607h56m1.712751788s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841696295 (607h56m1.712779341s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842184876 (607h56m1.713267921s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842429782 (607h56m1.713512817s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843206401 (607h56m1.714289446s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843831721 (607h56m1.714914766s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.844407488 (607h56m1.715490523s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:03.540346597 (607h56m3.411429653s) [upload=532790]: Next returned nil segment Dec 30 23:25:03.540374801 (607h56m3.411457866s) [upload=532790]: Waiting for error group managing segments... Dec 30 23:25:03.824335882 (607h56m3.695418927s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.832981394 (607h56m3.704064459s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.849417580 (607h56m3.720500625s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.866091639 (607h56m3.737174684s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.893000355 (607h56m3.764083401s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.929223007 (607h56m3.800306052s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.930027409 (607h56m3.801110454s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.026266778 (607h56m3.897349824s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.142674381 (607h56m4.013757426s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.156619473 (607h56m4.027702518s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.177527193 (607h56m4.048610238s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.249650634 (607h56m4.120733679s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320523523 (607h56m4.191606568s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320604697 (607h56m4.191687752s) [seg_pos={0 0}, upload=532790]: Segment reached optimal threshold of 13 pieces. Dec 30 23:25:04.321090162 (607h56m4.192173237s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:55884->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 23:25:04.321102807 (607h56m4.192185852s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321187618 (607h56m4.192270663s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection Dec 30 23:25:04.321199440 (607h56m4.192282485s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321436391 (607h56m4.192519446s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 23:25:04.321446772 (607h56m4.192529817s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321473181 (607h56m4.192556226s) [seg_pos={0 0}, upload=532790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:25:04.321595024 (607h56m4.192678069s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:25:04.321604051 (607h56m4.192687096s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:25:04.614658617 (607h56m4.485741662s) [upload=532790]: Done uploading segments... Dec 30 23:25:04.614685839 (607h56m4.485768884s) [upload=532790]: Upload finished. err: <nil> Duration: 5.864219509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 15.798615354s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:25:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.798634551s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)