Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...............X.X.XX.XX..XX.........X..X......X............] duration: 5.985284667s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.985301718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [................X.X.XX.XX..XX.........X..X......X...........] duration: 16.072457763s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:41:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 16.072487261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................X.X.XX.XX..XX.........X..X......X..........] duration: 5.883338427s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:40:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.883383993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................X.X.XX.XX..XX.........X..X......X.........] duration: 13.733484289s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:40:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 13.733526711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 31.875588608s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:40:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 31.875618374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 15.380793092s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.380821968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...................X.X.XX.XX..XX.........X..X......X........] duration: 6.138856336s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.138882646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 38.837139679s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:39:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 38.837154868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 29.322263381s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 29.322291165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....................X.X.XX.XX..XX.........X..X......X.......] duration: 6.654261158s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.654303901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 22.442474164s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:37:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 22.442500884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 23.506219681s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 23.506244928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....................X.X.XX.XX..XX.........X..X......X......] duration: 6.26271003s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:37:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.262740399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....................X.X.XX.XX..XX.........X..X......X.....] duration: 6.657799138s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 6.657829066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....................X.X.XX.XX..XX.........X..X......X....] duration: 5.977189669s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:36:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.977211322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.253536264s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.253561892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.....................X.X.XX.XX..XX.........X..X......X...] duration: 11.850903285s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 11.85092211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 20.83510189s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:35:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 20.835118801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 29.838482558s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:35:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 29.838510562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X.....................X.X.XX.XX..XX.........X..X......X..] duration: 7.67187508s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:34:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 7.671900107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 16.119217861s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 16.119241997s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X.....................X.X.XX.XX..XX.........X..X......X.] duration: 5.930689288s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:33:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 5.930713205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X.....................X.X.XX.XX..XX.........X..X......X] duration: 30.003301347s trace: https://jaeger.dev.storj.io/trace/272888c6112d1817 FAIL (2025-12-30 23:33:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ) context deadline exceeded ======LOGS====== Dec 30 23:33:09.066531096 (608h4m8.937614141s) [upload=532905]: Starting upload Dec 30 23:33:09.066614594 (608h4m8.937697629s) [upload=532905]: Done starting upload Dec 30 23:33:09.066707001 (608h4m8.937790036s) [upload=532905]: Uploading segments... Dec 30 23:33:09.066877676 (608h4m8.937960711s) [upload=532905]: Got next segment. Inline: false Dec 30 23:33:09.066986022 (608h4m8.938069057s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:33:09.066991813 (608h4m8.938074848s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:33:09.312619151 (608h4m9.183702186s) [seg_pos={0 0}, upload=532905]: Begin upload segment... Dec 30 23:33:09.312673304 (608h4m9.183756349s) [seg_pos={0 0}, upload=532905]: Done begin upload segment. Dec 30 23:33:09.312711998 (608h4m9.183795033s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.312760430 (608h4m9.183843475s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.313537019 (608h4m9.184620064s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314035939 (608h4m9.185118974s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.314471658 (608h4m9.185554703s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315009152 (608h4m9.186092187s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.315457015 (608h4m9.18654006s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.316471807 (608h4m9.187554842s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317125922 (608h4m9.188208957s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.317884767 (608h4m9.188967802s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.319217885 (608h4m9.19030092s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.320494126 (608h4m9.191577171s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.323071944 (608h4m9.194154989s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.324811266 (608h4m9.195894301s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.326227753 (608h4m9.197310798s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:09.327873847 (608h4m9.198956882s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Uploading piece... Dec 30 23:33:10.933323187 (608h4m10.804406232s) [upload=532905]: Next returned nil segment Dec 30 23:33:10.933339668 (608h4m10.804422713s) [upload=532905]: Waiting for error group managing segments... Dec 30 23:33:11.292342614 (608h4m11.163425669s) [noise=true , address=145.79.227.3:4150 , piece_id=GBWFFWHCKY3GTK2QN3WXUCBW3AHTOWE7PM4EGXINCWFWYNVL7B5Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.313263649 (608h4m11.184346684s) [noise=true , address=145.79.227.17:4971 , piece_id=ZOVDZ3TTTMITVAGS33FLJJSQBEAER5UYKUY3OUWOG7VHQVGHWEBA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.325503075 (608h4m11.19658612s) [noise=true , address=145.79.227.14:4838 , piece_id=GOTWTY5WAJSRQEXPJ2O2QIWBO5DEVKFRDSUWB667YSG76KH74YXQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.349178455 (608h4m11.2202615s) [noise=true , address=145.79.227.32:4834 , piece_id=ZE6IHEHLOOFQXMYZCD2T7JL67YMSIYXAK7PNLRXUUHX5V4J74OWA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.384075663 (608h4m11.255158708s) [noise=true , address=145.79.227.12:4765 , piece_id=4QOY2NXHF5UM7IUE7BPMDKDS24USS6GVYLKGAU6MLYCWHXOVKK6A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.388447396 (608h4m11.259530432s) [noise=true , address=145.79.227.9:4365 , piece_id=TM4GIECUWNMYCKCPEG7TMX4G3RZUVAVZ2ISPNGP7XXQ7RBH7XM3Q, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.391931531 (608h4m11.263014566s) [noise=true , address=145.79.227.18:4006 , piece_id=SCL2UB6OGNQM76L3GXSO5WA76WAS4ZPBAE7QFQOBAJL7EJPLVZYA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.412634842 (608h4m11.283717877s) [noise=true , address=145.79.227.47:4265 , piece_id=6DO264FX6QJZNEYUABSBNKT2PZ7JOV4TM37FQIWW3HAYMMWLDAHA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.551323619 (608h4m11.422406654s) [noise=true , address=77.90.31.21:4878 , piece_id=U7E3FDL6Z7LVPLQGVLZ3MMNUPDVCLKEV3KK2RUHHN2XDRYPHOFFQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.555828527 (608h4m11.426911562s) [noise=true , address=77.90.31.11:4555 , piece_id=RF2VGQEYHKFJ4X2X6YEFQKEOPKRXUFDIRMTNC4AKN2A4X7LCHUNQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.616790403 (608h4m11.487873449s) [noise=true , address=45.91.116.13:4154 , piece_id=AEZGJ6YZZO6GHVMIKEEPGTWWX55LXJUU2DLUF3JO4AYRHX3NZTBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:11.686823995 (608h4m11.55790703s) [noise=true , address=45.91.116.11:4119 , piece_id=LV6B5PDSB34ETFZMDIIQWQFESS2NLID2TTOHXJYBZ5LJ7MZPNWOQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704451985 (608h4m12.57553503s) [noise=true , address=45.91.116.9:4262 , piece_id=ALPIIYQUVWVOIPDWV4Z6QP3CMI4PC4L6AEBKPRFBR7XPDTEQHBMQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: <nil> Dec 30 23:33:12.704477804 (608h4m12.57556084s) [seg_pos={0 0}, upload=532905]: Segment reached optimal threshold of 13 pieces. Dec 30 23:33:12.704720186 (608h4m12.575803221s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704728842 (608h4m12.575811887s) [noise=true , address=145.79.227.33:4869 , piece_id=DJNZIMXIUCEWASJ6U7ZC2JLV5OSTVQOEQRVE2FTNMDV2B2U6XKBQ, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704913184 (608h4m12.575996219s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704924896 (608h4m12.576007931s) [noise=true , address=77.90.31.10:4518 , piece_id=UENWKZTNBNVJ65XU77G2Z7WOIWTIS6EYEPFFOAD43XBQGRUAQA3A, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704938261 (608h4m12.576021306s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 23:33:12.704944333 (608h4m12.576027368s) [noise=true , address=145.79.227.40:4051 , piece_id=TAWMRFVML76LVUHL7IUFXVRXIL23A3DLMCHSBPHP4SN2LVTUOQQA, seg_pos={0 0}, upload=532905]: Long tail cancellation due to optimal threshold Dec 30 23:33:12.704963069 (608h4m12.576046104s) [seg_pos={0 0}, upload=532905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:33:12.705090542 (608h4m12.576173587s) [upload=532905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:33:12.705097344 (608h4m12.576180389s) [upload=532905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:33:12.992700327 (608h4m12.863783362s) [upload=532905]: Done uploading segments... Dec 30 23:33:12.992725345 (608h4m12.86380838s) [upload=532905]: Upload finished. err: <nil> Duration: 30.00332417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 43.863959513s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:33:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 43.863981594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 48.280758753s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:32:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 48.280786384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.X.....................X.X.XX.XX..XX.........X..X......] duration: 6.865869969s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:32:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 6.865895947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X.....................X.X.XX.XX..XX.........X..X.....] duration: 15.889344776s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:31:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 15.889376545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 19.315281081s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 19.315310968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.X.....................X.X.XX.XX..XX.........X..X....] duration: 7.106070397s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:30:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 7.106086679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X.X.....................X.X.XX.XX..XX.........X..X...] duration: 28.928554418s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:30:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 28.928579967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.31524994s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:30:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.315267684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 25.145206296s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:29:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 25.145222647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X.X.....................X.X.XX.XX..XX.........X..X..] duration: 5.927783616s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:29:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 5.927810556s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 31.633212026s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:29:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 31.633237845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X........X.X.....................X.X.XX.XX..XX.........X..X.] duration: 6.029026756s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:28:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 6.029073685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 36.198460932s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:28:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 36.198496299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X........X.X.....................X.X.XX.XX..XX.........X..X] duration: 30.00252539s trace: https://jaeger.dev.storj.io/trace/d854cb9f7b591f6 FAIL (2025-12-30 23:27:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 23:27:07.963429233 (607h58m7.834512278s) [upload=532823]: Starting upload Dec 30 23:27:07.963527751 (607h58m7.834610786s) [upload=532823]: Done starting upload Dec 30 23:27:07.963615138 (607h58m7.834698173s) [upload=532823]: Uploading segments... Dec 30 23:27:07.963770714 (607h58m7.834853749s) [upload=532823]: Got next segment. Inline: false Dec 30 23:27:07.963826350 (607h58m7.834909395s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:27:07.963830899 (607h58m7.834913944s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:27:08.208197508 (607h58m8.079280543s) [seg_pos={0 0}, upload=532823]: Begin upload segment... Dec 30 23:27:08.208262761 (607h58m8.079345796s) [seg_pos={0 0}, upload=532823]: Done begin upload segment. Dec 30 23:27:08.208276819 (607h58m8.079359854s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208299412 (607h58m8.079382457s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.208837005 (607h58m8.07992005s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209262746 (607h58m8.080345791s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.209694117 (607h58m8.080777162s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210186966 (607h58m8.081270011s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.210626362 (607h58m8.081709397s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.211863608 (607h58m8.082946653s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.214571484 (607h58m8.085654529s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216581492 (607h58m8.087664527s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.216668387 (607h58m8.087751422s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.219567878 (607h58m8.090650913s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.220590175 (607h58m8.09167322s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.221436296 (607h58m8.092519341s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.223592952 (607h58m8.094675987s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:08.226159920 (607h58m8.097242955s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Uploading piece... Dec 30 23:27:09.922135570 (607h58m9.793218605s) [upload=532823]: Next returned nil segment Dec 30 23:27:09.922163724 (607h58m9.793246759s) [upload=532823]: Waiting for error group managing segments... Dec 30 23:27:10.209670967 (607h58m10.080754002s) [noise=true , address=145.79.227.18:4006 , piece_id=JDP2RU2HAOWE5GN2HSWECR2PUMC62U64LR6CNH74VRVJ7IVH5L6Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.210550622 (607h58m10.081633667s) [noise=true , address=145.79.227.3:4149 , piece_id=T2URF36X37CPXQD2U4D5UHYO3F2M3IA6QIFZJMVSNJRH7VBT3RFA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.219460136 (607h58m10.090543172s) [noise=true , address=145.79.227.19:4078 , piece_id=O3AL6OIMM4R6V6NTHLGZDNGOHP2YGLU3LSON7FJPQ7ZG76DG44WA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.227771692 (607h58m10.098854727s) [noise=true , address=145.79.227.11:4730 , piece_id=PW6IYVOFMCLFEPMUYGA3O6WQ6TXPUWSK57R37Q3RP6QIVLYCWF3Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.234384725 (607h58m10.10546776s) [noise=true , address=145.79.227.7:4295 , piece_id=55NTVPPPG7K4S6YGERWBFJ6EFNKRB44MPJ5UILGMEY7O3BP5MVHA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.235531198 (607h58m10.106614233s) [noise=true , address=145.79.227.42:4123 , piece_id=EZAOHBPFOSMW75FAMEFRNDNIK4VZUXKZ74UF7NGFKDIOZDJI5ZAA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.240430456 (607h58m10.111513491s) [noise=true , address=145.79.227.51:4410 , piece_id=T2DLGL477RFAG2SM2HXUDSLQU4REBVCYQKPMOYYIOQAEL2EHAHJA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.289350205 (607h58m10.16043324s) [noise=true , address=145.79.227.9:4367 , piece_id=OYJAX54IYKJYWESWFSVTOWCRPAW2BQBN5CXWVQ6BY7RQUPMD5R4Q, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.291107059 (607h58m10.162190125s) [noise=true , address=145.79.227.24:4582 , piece_id=6TFBZQBH6HUJF45EBQOTMQ5PAXDTSKW47PLSWZDOY4SIPFRYO3RA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.373303110 (607h58m10.244386145s) [noise=true , address=77.90.31.18:4808 , piece_id=B2343WVEYUSTDJRJBGUQNPHO3GBGKTOA7JCZYUH43G6N3OGDU3RQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.384944406 (607h58m10.256027441s) [noise=true , address=77.90.31.12:4589 , piece_id=PIFNKTGKSDZPTUKGFS64EM7CKAATPAJ6XK7OBL3F7H3KCW47EARA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.388459449 (607h58m10.259542484s) [noise=true , address=77.90.31.16:4736 , piece_id=BJ56SD4FAW7TGKWQ53JA5ERMBBGXZ242IXZ3L3RS2IV2ZBKID7BQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524623257 (607h58m10.395706303s) [noise=true , address=45.91.116.10:4082 , piece_id=NARHDXTUNDJTW4VMPMU4HWU25J7X74LJSG46KA47V5ZOCELXN3FQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: <nil> Dec 30 23:27:10.524643116 (607h58m10.395726161s) [seg_pos={0 0}, upload=532823]: Segment reached optimal threshold of 13 pieces. Dec 30 23:27:10.524843236 (607h58m10.395926291s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524850601 (607h58m10.395933646s) [noise=true , address=45.91.116.14:4189 , piece_id=BZA7AZ6GFHKCZZVSI6ZFJ4MAMECH5EUNYQQSMNE57ME4MV7HAAGA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524876610 (607h58m10.395959655s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524887241 (607h58m10.395970276s) [noise=true , address=145.79.227.33:4870 , piece_id=FNMSF5LFM3ICBOWW5VRMM4PILNXNWJHVAFEJDUHFNMVJ2ZO7SMCA, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.524982441 (607h58m10.396065476s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 30 23:27:10.524989395 (607h58m10.39607245s) [noise=true , address=45.91.116.5:4299 , piece_id=EFGELZ33V3KWYB2JLYFXW65ILLYKQH2S373NNNAGNWH5JUNKWMLQ, seg_pos={0 0}, upload=532823]: Long tail cancellation due to optimal threshold Dec 30 23:27:10.525004884 (607h58m10.396087919s) [seg_pos={0 0}, upload=532823]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:27:10.525077422 (607h58m10.396160467s) [upload=532823]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:27:10.525083023 (607h58m10.396166068s) [upload=532823]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:27:10.813922973 (607h58m10.685006029s) [upload=532823]: Done uploading segments... Dec 30 23:27:10.813944094 (607h58m10.685027139s) [upload=532823]: Upload finished. err: <nil> Duration: 30.002547963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 18.353761448s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:27:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 18.35379405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 23.730632257s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:26:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 23.730662856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X........X.X.....................X.X.XX.XX..XX.........X..] duration: 9.039778361s trace: https://jaeger.dev.storj.io/trace/5df0eedd8f155eba FAIL (2025-12-30 23:26:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ) context deadline exceeded ======LOGS====== d=532790]: Starting upload Dec 30 23:25:01.585730368 (607h56m1.456813413s) [upload=532790]: Done starting upload Dec 30 23:25:01.585844787 (607h56m1.456927832s) [upload=532790]: Uploading segments... Dec 30 23:25:01.586055338 (607h56m1.457138383s) [upload=532790]: Got next segment. Inline: false Dec 30 23:25:01.586138035 (607h56m1.45722108s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:25:01.586148915 (607h56m1.45723197s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:25:01.836805332 (607h56m1.707888387s) [seg_pos={0 0}, upload=532790]: Begin upload segment... Dec 30 23:25:01.836872830 (607h56m1.707955875s) [seg_pos={0 0}, upload=532790]: Done begin upload segment. Dec 30 23:25:01.836915182 (607h56m1.707998227s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839532234 (607h56m1.71061529s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839649688 (607h56m1.710732743s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839713740 (607h56m1.710796785s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840244561 (607h56m1.711327626s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840611871 (607h56m1.711694916s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840726599 (607h56m1.711809644s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841171677 (607h56m1.712254722s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841199178 (607h56m1.712282223s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841668743 (607h56m1.712751788s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841696295 (607h56m1.712779341s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842184876 (607h56m1.713267921s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842429782 (607h56m1.713512817s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843206401 (607h56m1.714289446s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843831721 (607h56m1.714914766s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.844407488 (607h56m1.715490523s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:03.540346597 (607h56m3.411429653s) [upload=532790]: Next returned nil segment Dec 30 23:25:03.540374801 (607h56m3.411457866s) [upload=532790]: Waiting for error group managing segments... Dec 30 23:25:03.824335882 (607h56m3.695418927s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.832981394 (607h56m3.704064459s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.849417580 (607h56m3.720500625s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.866091639 (607h56m3.737174684s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.893000355 (607h56m3.764083401s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.929223007 (607h56m3.800306052s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.930027409 (607h56m3.801110454s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.026266778 (607h56m3.897349824s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.142674381 (607h56m4.013757426s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.156619473 (607h56m4.027702518s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.177527193 (607h56m4.048610238s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.249650634 (607h56m4.120733679s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320523523 (607h56m4.191606568s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320604697 (607h56m4.191687752s) [seg_pos={0 0}, upload=532790]: Segment reached optimal threshold of 13 pieces. Dec 30 23:25:04.321090162 (607h56m4.192173237s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:55884->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 23:25:04.321102807 (607h56m4.192185852s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321187618 (607h56m4.192270663s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection Dec 30 23:25:04.321199440 (607h56m4.192282485s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321436391 (607h56m4.192519446s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 23:25:04.321446772 (607h56m4.192529817s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321473181 (607h56m4.192556226s) [seg_pos={0 0}, upload=532790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:25:04.321595024 (607h56m4.192678069s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:25:04.321604051 (607h56m4.192687096s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:25:04.614658617 (607h56m4.485741662s) [upload=532790]: Done uploading segments... Dec 30 23:25:04.614685839 (607h56m4.485768884s) [upload=532790]: Upload finished. err: <nil> Duration: 9.039807655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X........X.X.....................X.X.XX.XX..XX.........X.] duration: 5.864190944s trace: https://jaeger.dev.storj.io/trace/5df0eedd8f155eba FAIL (2025-12-30 23:25:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ) context deadline exceeded ======LOGS====== d=532790]: Starting upload Dec 30 23:25:01.585730368 (607h56m1.456813413s) [upload=532790]: Done starting upload Dec 30 23:25:01.585844787 (607h56m1.456927832s) [upload=532790]: Uploading segments... Dec 30 23:25:01.586055338 (607h56m1.457138383s) [upload=532790]: Got next segment. Inline: false Dec 30 23:25:01.586138035 (607h56m1.45722108s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:25:01.586148915 (607h56m1.45723197s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:25:01.836805332 (607h56m1.707888387s) [seg_pos={0 0}, upload=532790]: Begin upload segment... Dec 30 23:25:01.836872830 (607h56m1.707955875s) [seg_pos={0 0}, upload=532790]: Done begin upload segment. Dec 30 23:25:01.836915182 (607h56m1.707998227s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839532234 (607h56m1.71061529s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839649688 (607h56m1.710732743s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839713740 (607h56m1.710796785s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840244561 (607h56m1.711327626s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840611871 (607h56m1.711694916s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840726599 (607h56m1.711809644s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841171677 (607h56m1.712254722s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841199178 (607h56m1.712282223s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841668743 (607h56m1.712751788s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841696295 (607h56m1.712779341s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842184876 (607h56m1.713267921s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842429782 (607h56m1.713512817s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843206401 (607h56m1.714289446s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843831721 (607h56m1.714914766s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.844407488 (607h56m1.715490523s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:03.540346597 (607h56m3.411429653s) [upload=532790]: Next returned nil segment Dec 30 23:25:03.540374801 (607h56m3.411457866s) [upload=532790]: Waiting for error group managing segments... Dec 30 23:25:03.824335882 (607h56m3.695418927s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.832981394 (607h56m3.704064459s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.849417580 (607h56m3.720500625s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.866091639 (607h56m3.737174684s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.893000355 (607h56m3.764083401s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.929223007 (607h56m3.800306052s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.930027409 (607h56m3.801110454s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.026266778 (607h56m3.897349824s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.142674381 (607h56m4.013757426s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.156619473 (607h56m4.027702518s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.177527193 (607h56m4.048610238s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.249650634 (607h56m4.120733679s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320523523 (607h56m4.191606568s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320604697 (607h56m4.191687752s) [seg_pos={0 0}, upload=532790]: Segment reached optimal threshold of 13 pieces. Dec 30 23:25:04.321090162 (607h56m4.192173237s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:55884->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 23:25:04.321102807 (607h56m4.192185852s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321187618 (607h56m4.192270663s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection Dec 30 23:25:04.321199440 (607h56m4.192282485s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321436391 (607h56m4.192519446s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 23:25:04.321446772 (607h56m4.192529817s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321473181 (607h56m4.192556226s) [seg_pos={0 0}, upload=532790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:25:04.321595024 (607h56m4.192678069s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:25:04.321604051 (607h56m4.192687096s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:25:04.614658617 (607h56m4.485741662s) [upload=532790]: Done uploading segments... Dec 30 23:25:04.614685839 (607h56m4.485768884s) [upload=532790]: Upload finished. err: <nil> Duration: 5.864219509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 15.798615354s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:25:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 15.798634551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X........X.X.....................X.X.XX.XX..XX.........X] duration: 30.003717692s trace: https://jaeger.dev.storj.io/trace/5df0eedd8f155eba FAIL (2025-12-30 23:25:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ) context deadline exceeded ======LOGS====== d=532790]: Starting upload Dec 30 23:25:01.585730368 (607h56m1.456813413s) [upload=532790]: Done starting upload Dec 30 23:25:01.585844787 (607h56m1.456927832s) [upload=532790]: Uploading segments... Dec 30 23:25:01.586055338 (607h56m1.457138383s) [upload=532790]: Got next segment. Inline: false Dec 30 23:25:01.586138035 (607h56m1.45722108s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:25:01.586148915 (607h56m1.45723197s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:25:01.836805332 (607h56m1.707888387s) [seg_pos={0 0}, upload=532790]: Begin upload segment... Dec 30 23:25:01.836872830 (607h56m1.707955875s) [seg_pos={0 0}, upload=532790]: Done begin upload segment. Dec 30 23:25:01.836915182 (607h56m1.707998227s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839532234 (607h56m1.71061529s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839649688 (607h56m1.710732743s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.839713740 (607h56m1.710796785s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840244561 (607h56m1.711327626s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840611871 (607h56m1.711694916s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.840726599 (607h56m1.711809644s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841171677 (607h56m1.712254722s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841199178 (607h56m1.712282223s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841668743 (607h56m1.712751788s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.841696295 (607h56m1.712779341s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842184876 (607h56m1.713267921s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.842429782 (607h56m1.713512817s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843206401 (607h56m1.714289446s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.843831721 (607h56m1.714914766s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:01.844407488 (607h56m1.715490523s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Uploading piece... Dec 30 23:25:03.540346597 (607h56m3.411429653s) [upload=532790]: Next returned nil segment Dec 30 23:25:03.540374801 (607h56m3.411457866s) [upload=532790]: Waiting for error group managing segments... Dec 30 23:25:03.824335882 (607h56m3.695418927s) [noise=true , address=145.79.227.42:4122 , piece_id=JYPTMH6L6OCSOHJJGUUI3YBNQSBT7VKI5Q6ELN2SCXPX6XCWUQ6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.832981394 (607h56m3.704064459s) [noise=true , address=145.79.227.26:4654 , piece_id=77Y7UGBLSNP62HZEMNKRSRKNTEPRHS4TQ6YUCC2FYG4735W3XQTA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.849417580 (607h56m3.720500625s) [noise=true , address=145.79.227.37:4566 , piece_id=3HGMQQXDAPJRYWTQZQXAZ6SWDJDA2X424WYB3NRRZ5JMWORZ4I6A, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.866091639 (607h56m3.737174684s) [noise=true , address=145.79.227.39:4015 , piece_id=VPGUK2QFOSJQOMH6UUADX5GKFT3FCALK6ET5OGDP267AK55H2IEA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.893000355 (607h56m3.764083401s) [noise=true , address=145.79.227.52:4447 , piece_id=YE4O7QMIBO7IV4CKRJ7GC5GJHX2AUKMFR3LNUX4XGWMM4KQSFXIQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.929223007 (607h56m3.800306052s) [noise=true , address=145.79.227.23:4546 , piece_id=VCQ2325BNXTFXWC7EAGXRFD72WKPXY2BJ4HPQJJBH5TBLZBQGEDQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:03.930027409 (607h56m3.801110454s) [noise=true , address=145.79.227.3:4149 , piece_id=DSD4CDD2UUMMBRUKGVQVZTCTRYJQKDS7OTQ5AJOXHCGYUUED4KPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.026266778 (607h56m3.897349824s) [noise=true , address=77.90.31.11:4555 , piece_id=GZISGRGJOVBPRDS6CB5JIL2FRFQ746R6MOLTCLYSSJILPYCONNUQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.142674381 (607h56m4.013757426s) [noise=true , address=45.91.116.7:4011 , piece_id=2O5ACK7MOWG3NQIMMQ4OSJMTBDPJJAHIIVUEF4C4EGOKITPSHSKA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.156619473 (607h56m4.027702518s) [noise=true , address=45.91.116.8:4045 , piece_id=DAKOS7MLP6X4YXG5WUL4SU67AO6VOXDGGQJYGWH3VWOVC3KMCS2Q, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.177527193 (607h56m4.048610238s) [noise=true , address=45.91.116.12:4228 , piece_id=4OZDQDL2ARST5A4H55R3FG7WNW4VG23654BJOSW3DFB6FJBGFUFA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.249650634 (607h56m4.120733679s) [noise=true , address=45.91.116.9:4263 , piece_id=CJMS27D2EG2FAKLKXC2WRFCG35UWOCQDOAABH2A66HFIXGHVSOHQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320523523 (607h56m4.191606568s) [noise=true , address=45.91.116.5:4297 , piece_id=ARZI2MKJKQAQ4WPLR3GRY62PFEUPOVQ2ML2WIRIAOPW2OBT3XSVA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: <nil> Dec 30 23:25:04.320604697 (607h56m4.191687752s) [seg_pos={0 0}, upload=532790]: Segment reached optimal threshold of 13 pieces. Dec 30 23:25:04.321090162 (607h56m4.192173237s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:55884->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 23:25:04.321102807 (607h56m4.192185852s) [noise=true , address=77.90.31.12:4589 , piece_id=VRHLMGEM5PAR4XUQAMUHWUWEB3ACS3LEN6ANMB2KUHSE6GBJI2DA, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321187618 (607h56m4.192270663s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42218->77.90.31.21:4880: use of closed network connection Dec 30 23:25:04.321199440 (607h56m4.192282485s) [noise=true , address=77.90.31.21:4880 , piece_id=LTHZBO6LZXSJWHFJXA75T2YLYVXCC4WGV3SCA2TKUOT7DIQW3OCQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321436391 (607h56m4.192519446s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 23:25:04.321446772 (607h56m4.192529817s) [noise=true , address=145.79.227.35:4483 , piece_id=SSGBUUP25R7G4EWQ7B6EJOOJEOM7WT32DNJBJPNI4MYPHANCXOPQ, seg_pos={0 0}, upload=532790]: Long tail cancellation due to optimal threshold Dec 30 23:25:04.321473181 (607h56m4.192556226s) [seg_pos={0 0}, upload=532790]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:25:04.321595024 (607h56m4.192678069s) [upload=532790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:25:04.321604051 (607h56m4.192687096s) [upload=532790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:25:04.614658617 (607h56m4.485741662s) [upload=532790]: Done uploading segments... Dec 30 23:25:04.614685839 (607h56m4.485768884s) [upload=532790]: Upload finished. err: <nil> Duration: 30.003747981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...................X.......................................] duration: 22.316349619s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 22.316375778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................X......................................] duration: 22.848441404s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:24:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 22.848465179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.X........X.X.....................X.X.XX.XX..XX.........] duration: 26.21568979s trace: https://jaeger.dev.storj.io/trace/344a5e3879f819fd FAIL (2025-12-30 23:24:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 23:17:26.163686753 (607h48m26.034769788s) [upload=532678]: Starting upload Dec 30 23:17:26.163732611 (607h48m26.034815646s) [upload=532678]: Done starting upload Dec 30 23:17:26.163819707 (607h48m26.034902742s) [upload=532678]: Uploading segments... Dec 30 23:17:26.163990552 (607h48m26.035073587s) [upload=532678]: Got next segment. Inline: false Dec 30 23:17:26.164027803 (607h48m26.035110838s) [upload=532678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:17:26.164033864 (607h48m26.035116909s) [upload=532678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:17:26.409423802 (607h48m26.280506847s) [seg_pos={0 0}, upload=532678]: Begin upload segment... Dec 30 23:17:26.409490308 (607h48m26.280573344s) [seg_pos={0 0}, upload=532678]: Done begin upload segment. Dec 30 23:17:26.409517400 (607h48m26.280600435s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.409898296 (607h48m26.280981341s) [noise=true , address=145.79.227.27:4691 , piece_id=ESAS3I4KB5SMZTSFAEO2HT34X4ZCOVLLH3UUYG4WZSGL7742APDA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410350807 (607h48m26.281433842s) [noise=true , address=145.79.227.46:4601 , piece_id=APHUZ6IR3BN7TSBMPXDHWLXU5ZTYVBWC2DTYYEZI4H5MOFBL2SMA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410390623 (607h48m26.281473658s) [noise=true , address=45.91.116.5:4298 , piece_id=F7JGPVWFVFM4EH53PNOF3FN545QBMSWXWFSPD4T66WTKEWH7P6EQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410648223 (607h48m26.281731258s) [noise=true , address=145.79.227.38:4977 , piece_id=JG3BMJTNQOWZSRL5NIQRVYJ3NZD52KWYRR6SVPUU4OOAGA2UOGIQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410819630 (607h48m26.281902665s) [noise=true , address=45.91.116.8:4047 , piece_id=ETHMZ5M75UFOQI3V6JWB4AESKPPTFMKXS34OQZBGRKQQXNXUDQQQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410967250 (607h48m26.282050285s) [noise=true , address=145.79.227.21:4473 , piece_id=LADDXLGX5QUWXW45L45PMXAJ6YCDZKINKLX7OQEYKMARGUDIGMZQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411174455 (607h48m26.28225749s) [noise=true , address=145.79.227.45:4230 , piece_id=KL42IWYGJHNCKFZCCVNEFVCIDJZJGLOXA6QFVMKASCWTUVPJCSPQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411404744 (607h48m26.282487779s) [noise=true , address=145.79.227.6:4259 , piece_id=4S33APW3HLF4BRX7DIWDY7LQNPYMFJPTAHOBSVJWFFFQZMRRUJGQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411542877 (607h48m26.282625912s) [noise=true , address=145.79.227.5:4223 , piece_id=D5QHM7HOKJKCZBFKG6W5YJVHUG5PAFWJLP2YV6AKTBITAVVCVKTA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411735233 (607h48m26.282818278s) [noise=true , address=145.79.227.32:4835 , piece_id=W3CNTIHNMDP5KKFDHY44T4A72MUVPUYSA3RXJ5HOF7PTJONQSVCA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411845773 (607h48m26.282928808s) [noise=true , address=145.79.227.3:4150 , piece_id=6ISAZ3UPMAKWTCWPNOBBK7IW3EVTA3FX3SVKGGEFUEVXCJNJB7VA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412031628 (607h48m26.283114663s) [noise=true , address=77.90.31.11:4556 , piece_id=EPOAJUQL5UAFVQH4L4S3HX4WR5BDGDR74DPH25HYUQRYEGSL4VQQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412315207 (607h48m26.283398253s) [noise=true , address=145.79.227.12:4765 , piece_id=X5X4PKN3TCLBIRKZBZ5L5CE4EBP7ZYAEPHW5YOWDXKHTREP5OK4Q, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412360834 (607h48m26.283443869s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412808116 (607h48m26.283891151s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:28.099853741 (607h48m27.970936776s) [upload=532678]: Next returned nil segment Dec 30 23:17:28.099867989 (607h48m27.970951024s) [upload=532678]: Waiting for error group managing segments... Dec 30 23:17:28.385604178 (607h48m28.256687223s) [noise=true , address=145.79.227.21:4473 , piece_id=LADDXLGX5QUWXW45L45PMXAJ6YCDZKINKLX7OQEYKMARGUDIGMZQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.388567731 (607h48m28.259650766s) [noise=true , address=145.79.227.6:4259 , piece_id=4S33APW3HLF4BRX7DIWDY7LQNPYMFJPTAHOBSVJWFFFQZMRRUJGQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.395729098 (607h48m28.266812133s) [noise=true , address=145.79.227.3:4150 , piece_id=6ISAZ3UPMAKWTCWPNOBBK7IW3EVTA3FX3SVKGGEFUEVXCJNJB7VA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.404998658 (607h48m28.276081693s) [noise=true , address=145.79.227.45:4230 , piece_id=KL42IWYGJHNCKFZCCVNEFVCIDJZJGLOXA6QFVMKASCWTUVPJCSPQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.409729927 (607h48m28.280812962s) [noise=true , address=145.79.227.12:4765 , piece_id=X5X4PKN3TCLBIRKZBZ5L5CE4EBP7ZYAEPHW5YOWDXKHTREP5OK4Q, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.412631772 (607h48m28.283714807s) [noise=true , address=145.79.227.46:4601 , piece_id=APHUZ6IR3BN7TSBMPXDHWLXU5ZTYVBWC2DTYYEZI4H5MOFBL2SMA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.413005193 (607h48m28.284088229s) [noise=true , address=145.79.227.32:4835 , piece_id=W3CNTIHNMDP5KKFDHY44T4A72MUVPUYSA3RXJ5HOF7PTJONQSVCA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.417313496 (607h48m28.288396532s) [noise=true , address=145.79.227.5:4223 , piece_id=D5QHM7HOKJKCZBFKG6W5YJVHUG5PAFWJLP2YV6AKTBITAVVCVKTA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.479149799 (607h48m28.350232844s) [noise=true , address=145.79.227.27:4691 , piece_id=ESAS3I4KB5SMZTSFAEO2HT34X4ZCOVLLH3UUYG4WZSGL7742APDA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.482390901 (607h48m28.353473936s) [noise=true , address=145.79.227.38:4977 , piece_id=JG3BMJTNQOWZSRL5NIQRVYJ3NZD52KWYRR6SVPUU4OOAGA2UOGIQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.702512013 (607h48m28.573595048s) [noise=true , address=77.90.31.11:4556 , piece_id=EPOAJUQL5UAFVQH4L4S3HX4WR5BDGDR74DPH25HYUQRYEGSL4VQQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.718307509 (607h48m28.589390545s) [noise=true , address=45.91.116.5:4298 , piece_id=F7JGPVWFVFM4EH53PNOF3FN545QBMSWXWFSPD4T66WTKEWH7P6EQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.746171766 (607h48m28.617254811s) [noise=true , address=45.91.116.8:4047 , piece_id=ETHMZ5M75UFOQI3V6JWB4AESKPPTFMKXS34OQZBGRKQQXNXUDQQQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.746196272 (607h48m28.617279317s) [seg_pos={0 0}, upload=532678]: Segment reached optimal threshold of 13 pieces. Dec 30 23:17:28.746428685 (607h48m28.61751173s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection Dec 30 23:17:28.746437201 (607h48m28.617520246s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746563632 (607h48m28.617646677s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 23:17:28.746580754 (607h48m28.6176638s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746811374 (607h48m28.617894409s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 23:17:28.746819559 (607h48m28.617902604s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746838696 (607h48m28.617921741s) [seg_pos={0 0}, upload=532678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:17:28.746920381 (607h48m28.618003416s) [upload=532678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:17:28.746926503 (607h48m28.618009538s) [upload=532678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:17:29.037300995 (607h48m28.90838404s) [upload=532678]: Done uploading segments... Dec 30 23:17:29.037323738 (607h48m28.908406773s) [upload=532678]: Upload finished. err: <nil> Duration: 26.215713425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................X.....................................] duration: 31.052066969s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:24:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 31.052094041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.X........X.X.....................X.X.XX.XX..XX........] duration: 6.143995426s trace: https://jaeger.dev.storj.io/trace/344a5e3879f819fd FAIL (2025-12-30 23:23:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 23:17:26.163686753 (607h48m26.034769788s) [upload=532678]: Starting upload Dec 30 23:17:26.163732611 (607h48m26.034815646s) [upload=532678]: Done starting upload Dec 30 23:17:26.163819707 (607h48m26.034902742s) [upload=532678]: Uploading segments... Dec 30 23:17:26.163990552 (607h48m26.035073587s) [upload=532678]: Got next segment. Inline: false Dec 30 23:17:26.164027803 (607h48m26.035110838s) [upload=532678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:17:26.164033864 (607h48m26.035116909s) [upload=532678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:17:26.409423802 (607h48m26.280506847s) [seg_pos={0 0}, upload=532678]: Begin upload segment... Dec 30 23:17:26.409490308 (607h48m26.280573344s) [seg_pos={0 0}, upload=532678]: Done begin upload segment. Dec 30 23:17:26.409517400 (607h48m26.280600435s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.409898296 (607h48m26.280981341s) [noise=true , address=145.79.227.27:4691 , piece_id=ESAS3I4KB5SMZTSFAEO2HT34X4ZCOVLLH3UUYG4WZSGL7742APDA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410350807 (607h48m26.281433842s) [noise=true , address=145.79.227.46:4601 , piece_id=APHUZ6IR3BN7TSBMPXDHWLXU5ZTYVBWC2DTYYEZI4H5MOFBL2SMA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410390623 (607h48m26.281473658s) [noise=true , address=45.91.116.5:4298 , piece_id=F7JGPVWFVFM4EH53PNOF3FN545QBMSWXWFSPD4T66WTKEWH7P6EQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410648223 (607h48m26.281731258s) [noise=true , address=145.79.227.38:4977 , piece_id=JG3BMJTNQOWZSRL5NIQRVYJ3NZD52KWYRR6SVPUU4OOAGA2UOGIQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410819630 (607h48m26.281902665s) [noise=true , address=45.91.116.8:4047 , piece_id=ETHMZ5M75UFOQI3V6JWB4AESKPPTFMKXS34OQZBGRKQQXNXUDQQQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410967250 (607h48m26.282050285s) [noise=true , address=145.79.227.21:4473 , piece_id=LADDXLGX5QUWXW45L45PMXAJ6YCDZKINKLX7OQEYKMARGUDIGMZQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411174455 (607h48m26.28225749s) [noise=true , address=145.79.227.45:4230 , piece_id=KL42IWYGJHNCKFZCCVNEFVCIDJZJGLOXA6QFVMKASCWTUVPJCSPQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411404744 (607h48m26.282487779s) [noise=true , address=145.79.227.6:4259 , piece_id=4S33APW3HLF4BRX7DIWDY7LQNPYMFJPTAHOBSVJWFFFQZMRRUJGQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411542877 (607h48m26.282625912s) [noise=true , address=145.79.227.5:4223 , piece_id=D5QHM7HOKJKCZBFKG6W5YJVHUG5PAFWJLP2YV6AKTBITAVVCVKTA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411735233 (607h48m26.282818278s) [noise=true , address=145.79.227.32:4835 , piece_id=W3CNTIHNMDP5KKFDHY44T4A72MUVPUYSA3RXJ5HOF7PTJONQSVCA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411845773 (607h48m26.282928808s) [noise=true , address=145.79.227.3:4150 , piece_id=6ISAZ3UPMAKWTCWPNOBBK7IW3EVTA3FX3SVKGGEFUEVXCJNJB7VA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412031628 (607h48m26.283114663s) [noise=true , address=77.90.31.11:4556 , piece_id=EPOAJUQL5UAFVQH4L4S3HX4WR5BDGDR74DPH25HYUQRYEGSL4VQQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412315207 (607h48m26.283398253s) [noise=true , address=145.79.227.12:4765 , piece_id=X5X4PKN3TCLBIRKZBZ5L5CE4EBP7ZYAEPHW5YOWDXKHTREP5OK4Q, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412360834 (607h48m26.283443869s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412808116 (607h48m26.283891151s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:28.099853741 (607h48m27.970936776s) [upload=532678]: Next returned nil segment Dec 30 23:17:28.099867989 (607h48m27.970951024s) [upload=532678]: Waiting for error group managing segments... Dec 30 23:17:28.385604178 (607h48m28.256687223s) [noise=true , address=145.79.227.21:4473 , piece_id=LADDXLGX5QUWXW45L45PMXAJ6YCDZKINKLX7OQEYKMARGUDIGMZQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.388567731 (607h48m28.259650766s) [noise=true , address=145.79.227.6:4259 , piece_id=4S33APW3HLF4BRX7DIWDY7LQNPYMFJPTAHOBSVJWFFFQZMRRUJGQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.395729098 (607h48m28.266812133s) [noise=true , address=145.79.227.3:4150 , piece_id=6ISAZ3UPMAKWTCWPNOBBK7IW3EVTA3FX3SVKGGEFUEVXCJNJB7VA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.404998658 (607h48m28.276081693s) [noise=true , address=145.79.227.45:4230 , piece_id=KL42IWYGJHNCKFZCCVNEFVCIDJZJGLOXA6QFVMKASCWTUVPJCSPQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.409729927 (607h48m28.280812962s) [noise=true , address=145.79.227.12:4765 , piece_id=X5X4PKN3TCLBIRKZBZ5L5CE4EBP7ZYAEPHW5YOWDXKHTREP5OK4Q, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.412631772 (607h48m28.283714807s) [noise=true , address=145.79.227.46:4601 , piece_id=APHUZ6IR3BN7TSBMPXDHWLXU5ZTYVBWC2DTYYEZI4H5MOFBL2SMA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.413005193 (607h48m28.284088229s) [noise=true , address=145.79.227.32:4835 , piece_id=W3CNTIHNMDP5KKFDHY44T4A72MUVPUYSA3RXJ5HOF7PTJONQSVCA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.417313496 (607h48m28.288396532s) [noise=true , address=145.79.227.5:4223 , piece_id=D5QHM7HOKJKCZBFKG6W5YJVHUG5PAFWJLP2YV6AKTBITAVVCVKTA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.479149799 (607h48m28.350232844s) [noise=true , address=145.79.227.27:4691 , piece_id=ESAS3I4KB5SMZTSFAEO2HT34X4ZCOVLLH3UUYG4WZSGL7742APDA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.482390901 (607h48m28.353473936s) [noise=true , address=145.79.227.38:4977 , piece_id=JG3BMJTNQOWZSRL5NIQRVYJ3NZD52KWYRR6SVPUU4OOAGA2UOGIQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.702512013 (607h48m28.573595048s) [noise=true , address=77.90.31.11:4556 , piece_id=EPOAJUQL5UAFVQH4L4S3HX4WR5BDGDR74DPH25HYUQRYEGSL4VQQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.718307509 (607h48m28.589390545s) [noise=true , address=45.91.116.5:4298 , piece_id=F7JGPVWFVFM4EH53PNOF3FN545QBMSWXWFSPD4T66WTKEWH7P6EQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.746171766 (607h48m28.617254811s) [noise=true , address=45.91.116.8:4047 , piece_id=ETHMZ5M75UFOQI3V6JWB4AESKPPTFMKXS34OQZBGRKQQXNXUDQQQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.746196272 (607h48m28.617279317s) [seg_pos={0 0}, upload=532678]: Segment reached optimal threshold of 13 pieces. Dec 30 23:17:28.746428685 (607h48m28.61751173s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection Dec 30 23:17:28.746437201 (607h48m28.617520246s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746563632 (607h48m28.617646677s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 23:17:28.746580754 (607h48m28.6176638s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746811374 (607h48m28.617894409s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 23:17:28.746819559 (607h48m28.617902604s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746838696 (607h48m28.617921741s) [seg_pos={0 0}, upload=532678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:17:28.746920381 (607h48m28.618003416s) [upload=532678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:17:28.746926503 (607h48m28.618009538s) [upload=532678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:17:29.037300995 (607h48m28.90838404s) [upload=532678]: Done uploading segments... Dec 30 23:17:29.037323738 (607h48m28.908406773s) [upload=532678]: Upload finished. err: <nil> Duration: 6.144024803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X........X.X.....................X.X.XX.XX..XX.......] duration: 5.717781355s trace: https://jaeger.dev.storj.io/trace/344a5e3879f819fd FAIL (2025-12-30 23:23:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 23:17:26.163686753 (607h48m26.034769788s) [upload=532678]: Starting upload Dec 30 23:17:26.163732611 (607h48m26.034815646s) [upload=532678]: Done starting upload Dec 30 23:17:26.163819707 (607h48m26.034902742s) [upload=532678]: Uploading segments... Dec 30 23:17:26.163990552 (607h48m26.035073587s) [upload=532678]: Got next segment. Inline: false Dec 30 23:17:26.164027803 (607h48m26.035110838s) [upload=532678]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 23:17:26.164033864 (607h48m26.035116909s) [upload=532678]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 23:17:26.409423802 (607h48m26.280506847s) [seg_pos={0 0}, upload=532678]: Begin upload segment... Dec 30 23:17:26.409490308 (607h48m26.280573344s) [seg_pos={0 0}, upload=532678]: Done begin upload segment. Dec 30 23:17:26.409517400 (607h48m26.280600435s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.409898296 (607h48m26.280981341s) [noise=true , address=145.79.227.27:4691 , piece_id=ESAS3I4KB5SMZTSFAEO2HT34X4ZCOVLLH3UUYG4WZSGL7742APDA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410350807 (607h48m26.281433842s) [noise=true , address=145.79.227.46:4601 , piece_id=APHUZ6IR3BN7TSBMPXDHWLXU5ZTYVBWC2DTYYEZI4H5MOFBL2SMA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410390623 (607h48m26.281473658s) [noise=true , address=45.91.116.5:4298 , piece_id=F7JGPVWFVFM4EH53PNOF3FN545QBMSWXWFSPD4T66WTKEWH7P6EQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410648223 (607h48m26.281731258s) [noise=true , address=145.79.227.38:4977 , piece_id=JG3BMJTNQOWZSRL5NIQRVYJ3NZD52KWYRR6SVPUU4OOAGA2UOGIQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410819630 (607h48m26.281902665s) [noise=true , address=45.91.116.8:4047 , piece_id=ETHMZ5M75UFOQI3V6JWB4AESKPPTFMKXS34OQZBGRKQQXNXUDQQQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.410967250 (607h48m26.282050285s) [noise=true , address=145.79.227.21:4473 , piece_id=LADDXLGX5QUWXW45L45PMXAJ6YCDZKINKLX7OQEYKMARGUDIGMZQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411174455 (607h48m26.28225749s) [noise=true , address=145.79.227.45:4230 , piece_id=KL42IWYGJHNCKFZCCVNEFVCIDJZJGLOXA6QFVMKASCWTUVPJCSPQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411404744 (607h48m26.282487779s) [noise=true , address=145.79.227.6:4259 , piece_id=4S33APW3HLF4BRX7DIWDY7LQNPYMFJPTAHOBSVJWFFFQZMRRUJGQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411542877 (607h48m26.282625912s) [noise=true , address=145.79.227.5:4223 , piece_id=D5QHM7HOKJKCZBFKG6W5YJVHUG5PAFWJLP2YV6AKTBITAVVCVKTA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411735233 (607h48m26.282818278s) [noise=true , address=145.79.227.32:4835 , piece_id=W3CNTIHNMDP5KKFDHY44T4A72MUVPUYSA3RXJ5HOF7PTJONQSVCA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.411845773 (607h48m26.282928808s) [noise=true , address=145.79.227.3:4150 , piece_id=6ISAZ3UPMAKWTCWPNOBBK7IW3EVTA3FX3SVKGGEFUEVXCJNJB7VA, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412031628 (607h48m26.283114663s) [noise=true , address=77.90.31.11:4556 , piece_id=EPOAJUQL5UAFVQH4L4S3HX4WR5BDGDR74DPH25HYUQRYEGSL4VQQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412315207 (607h48m26.283398253s) [noise=true , address=145.79.227.12:4765 , piece_id=X5X4PKN3TCLBIRKZBZ5L5CE4EBP7ZYAEPHW5YOWDXKHTREP5OK4Q, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412360834 (607h48m26.283443869s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:26.412808116 (607h48m26.283891151s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Uploading piece... Dec 30 23:17:28.099853741 (607h48m27.970936776s) [upload=532678]: Next returned nil segment Dec 30 23:17:28.099867989 (607h48m27.970951024s) [upload=532678]: Waiting for error group managing segments... Dec 30 23:17:28.385604178 (607h48m28.256687223s) [noise=true , address=145.79.227.21:4473 , piece_id=LADDXLGX5QUWXW45L45PMXAJ6YCDZKINKLX7OQEYKMARGUDIGMZQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.388567731 (607h48m28.259650766s) [noise=true , address=145.79.227.6:4259 , piece_id=4S33APW3HLF4BRX7DIWDY7LQNPYMFJPTAHOBSVJWFFFQZMRRUJGQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.395729098 (607h48m28.266812133s) [noise=true , address=145.79.227.3:4150 , piece_id=6ISAZ3UPMAKWTCWPNOBBK7IW3EVTA3FX3SVKGGEFUEVXCJNJB7VA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.404998658 (607h48m28.276081693s) [noise=true , address=145.79.227.45:4230 , piece_id=KL42IWYGJHNCKFZCCVNEFVCIDJZJGLOXA6QFVMKASCWTUVPJCSPQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.409729927 (607h48m28.280812962s) [noise=true , address=145.79.227.12:4765 , piece_id=X5X4PKN3TCLBIRKZBZ5L5CE4EBP7ZYAEPHW5YOWDXKHTREP5OK4Q, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.412631772 (607h48m28.283714807s) [noise=true , address=145.79.227.46:4601 , piece_id=APHUZ6IR3BN7TSBMPXDHWLXU5ZTYVBWC2DTYYEZI4H5MOFBL2SMA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.413005193 (607h48m28.284088229s) [noise=true , address=145.79.227.32:4835 , piece_id=W3CNTIHNMDP5KKFDHY44T4A72MUVPUYSA3RXJ5HOF7PTJONQSVCA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.417313496 (607h48m28.288396532s) [noise=true , address=145.79.227.5:4223 , piece_id=D5QHM7HOKJKCZBFKG6W5YJVHUG5PAFWJLP2YV6AKTBITAVVCVKTA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.479149799 (607h48m28.350232844s) [noise=true , address=145.79.227.27:4691 , piece_id=ESAS3I4KB5SMZTSFAEO2HT34X4ZCOVLLH3UUYG4WZSGL7742APDA, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.482390901 (607h48m28.353473936s) [noise=true , address=145.79.227.38:4977 , piece_id=JG3BMJTNQOWZSRL5NIQRVYJ3NZD52KWYRR6SVPUU4OOAGA2UOGIQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.702512013 (607h48m28.573595048s) [noise=true , address=77.90.31.11:4556 , piece_id=EPOAJUQL5UAFVQH4L4S3HX4WR5BDGDR74DPH25HYUQRYEGSL4VQQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.718307509 (607h48m28.589390545s) [noise=true , address=45.91.116.5:4298 , piece_id=F7JGPVWFVFM4EH53PNOF3FN545QBMSWXWFSPD4T66WTKEWH7P6EQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.746171766 (607h48m28.617254811s) [noise=true , address=45.91.116.8:4047 , piece_id=ETHMZ5M75UFOQI3V6JWB4AESKPPTFMKXS34OQZBGRKQQXNXUDQQQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: <nil> Dec 30 23:17:28.746196272 (607h48m28.617279317s) [seg_pos={0 0}, upload=532678]: Segment reached optimal threshold of 13 pieces. Dec 30 23:17:28.746428685 (607h48m28.61751173s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37336->77.90.31.19:4841: use of closed network connection Dec 30 23:17:28.746437201 (607h48m28.617520246s) [noise=true , address=77.90.31.19:4841 , piece_id=3IAGUVDXRBQKEJ6DGQYNWW4NUYFV2THGJIB62NJ6MNYM335ZFDTQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746563632 (607h48m28.617646677s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 23:17:28.746580754 (607h48m28.6176638s) [noise=true , address=45.91.116.7:4012 , piece_id=NSBMVRSX4LG55PVXJIIGSZYKOE2XLIPGSZ2SZTDXLZEYPVSFZRXQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746811374 (607h48m28.617894409s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 23:17:28.746819559 (607h48m28.617902604s) [noise=true , address=45.91.116.10:4081 , piece_id=Z23WP63DY73WMQKEKZTHEVJ4A4DKN6343NAL5HRJ4UEI3367N6MQ, seg_pos={0 0}, upload=532678]: Long tail cancellation due to optimal threshold Dec 30 23:17:28.746838696 (607h48m28.617921741s) [seg_pos={0 0}, upload=532678]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 23:17:28.746920381 (607h48m28.618003416s) [upload=532678]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 23:17:28.746926503 (607h48m28.618009538s) [upload=532678]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 23:17:29.037300995 (607h48m28.90838404s) [upload=532678]: Done uploading segments... Dec 30 23:17:29.037323738 (607h48m28.908406773s) [upload=532678]: Upload finished. err: <nil> Duration: 5.717811623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...................X....................................] duration: 9.151023455s trace: https://jaeger.dev.storj.io/trace/36b8c702998585fe FAIL (2025-12-30 23:22:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097267582 (607h25m31.968350617s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.097626897 (607h25m31.968709932s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098065591 (607h25m31.969148626s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098463880 (607h25m31.969546915s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.098770363 (607h25m31.969853398s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099117705 (607h25m31.97020074s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.099661020 (607h25m31.970744055s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101322993 (607h25m31.972406029s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.101491064 (607h25m31.972574109s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.103074319 (607h25m31.974157354s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.104362412 (607h25m31.975445447s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.107739452 (607h25m31.978822487s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.110102461 (607h25m31.981185506s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.112342536 (607h25m31.983425571s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:32.113632713 (607h25m31.984715748s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Uploading piece... Dec 30 22:54:33.519095442 (607h25m33.390178477s) [upload=532346]: Next returned nil segment Dec 30 22:54:33.519108778 (607h25m33.390191823s) [upload=532346]: Waiting for error group managing segments... Dec 30 22:54:33.962988194 (607h25m33.834071229s) [noise=true , address=64.49.13.140:4509 , piece_id=4GK3YYBWV55HLUGDTE7S4N36M4YXNCL7BHS2NYEXZKZMX7XMDAXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:33.963210899 (607h25m33.834293934s) [noise=true , address=145.79.227.17:4974 , piece_id=V62DABSYAA7HZJ6JWH5GNDY6ST7W72AHXD7MW67WDAZXWC7ALORA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.004327659 (607h25m33.875410714s) [noise=true , address=145.79.227.22:4533 , piece_id=JT6DO2ARFI7TPITY4DN7RIO6OV2AMWJFOQP2HEG3SSJHDEQHOUOQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.126561594 (607h25m33.997644629s) [noise=true , address=77.90.31.19:4872 , piece_id=E34HX4QTU4K67SMONK3LKZSHJYD7AOU3QIP5BVWUIHS52B4QYMWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.137706816 (607h25m34.008789851s) [noise=true , address=77.90.31.18:4814 , piece_id=4UANYHKRQDVGKCMPVT5RA3ZVOJPA5YUKCAHUXEDTJ26UZ6G77IYA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.177004383 (607h25m34.048087418s) [noise=true , address=77.90.31.14:4667 , piece_id=P7XZYSM4NL4H5MEAE6SAVTE2ZVYOCH2XEHKRWECOWVFXVNZPCPXA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.199118650 (607h25m34.070201685s) [noise=true , address=77.90.31.17:4791 , piece_id=KWFGVQIW5LIRRV2OLPOC6UMV3TQ35E4V5JEFPGXQL62OZY5VVFIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256056906 (607h25m34.127139941s) [noise=true , address=77.90.31.15:4719 , piece_id=KAKZE6X6KT47XEXXHZZRC22F637UPSCBRQIBUFLWD6YZO6UZ66DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.256464702 (607h25m34.127547737s) [noise=true , address=77.90.31.12:4615 , piece_id=FK65ZOVDE33M6ZF5IFTOP7D4SVIH533ZAGHIWF4IQUXTBI4M6EQA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.434602764 (607h25m34.305685799s) [noise=true , address=77.90.31.21:4905 , piece_id=TLZUVJPIUJUE3U6VSZHJYUAH6BEHIPLN3E4SVMMKQWCR7HJ6B57A, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.852149863 (607h25m34.723232908s) [noise=true , address=145.79.227.30:4792 , piece_id=E3RFRJWDPN6U6S4QOJ5AXKEQFTRXB3KBYBE2RXRJPL7QWSUKK6DA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.888060369 (607h25m34.759143404s) [noise=true , address=193.149.164.143:4698 , piece_id=JGQAWKAKXDZUHZRHLH6IRLBSO234Z2424RXNSHSHPJKJ6ZO7ODWA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.893346294 (607h25m34.76442933s) [noise=true , address=145.79.227.44:4196 , piece_id=UMYI73K6C7TEM2KI3HCEBIGC4OOLFBI7375XYJCMDWRXCXBTEIIQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944460230 (607h25m34.815543265s) [noise=true , address=145.79.227.49:4346 , piece_id=EPB2QKBL2DDDVS7DFQEK5QCJBBHJJRJC3Z3VPRTS7J25D7JKGERA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: <nil> Dec 30 22:54:34.944482723 (607h25m34.815565768s) [seg_pos={0 0}, upload=532346]: Segment reached optimal threshold of 14 pieces. Dec 30 22:54:34.944653889 (607h25m34.815736924s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZjUHDRm2WAGu9XKeZeEF3sRqZF5bqJAVupVwTwSTEKQDhucKe): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944659610 (607h25m34.815742655s) [noise=true , address=45.91.116.7:4030 , piece_id=BFRFBPO6ZVHJCE4XKSFXMACIZRZI3M4SJHSKO5I2YIPXXUB62NOA, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944762977 (607h25m34.815846012s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944768116 (607h25m34.815851151s) [noise=true , address=145.79.227.40:4075 , piece_id=JZA4VCRRNZIIXO5SY3MCFOLLCXQAOLKNKNH74CMB63CFJAO2H2XQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944781973 (607h25m34.815865028s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36330->64.49.13.169:4706: use of closed network connection Dec 30 22:54:34.944792112 (607h25m34.815875147s) [noise=true , address=64.49.13.169:4706 , piece_id=JO3CS2LXEP63WXIV26DCZYPOWUTXB2KTQZUSKELTX7R6IGEBDE6Q, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944864049 (607h25m34.815947084s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 30 22:54:34.944869600 (607h25m34.815952645s) [noise=true , address=145.79.227.33:4899 , piece_id=BCZ66TH4IJJAB6QTVXDAL2HAJNJZIB42QBYLGURZA7I24ZONXJNQ, seg_pos={0 0}, upload=532346]: Long tail cancellation due to optimal threshold Dec 30 22:54:34.944899597 (607h25m34.815982632s) [seg_pos={0 0}, upload=532346]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 22:54:34.944981202 (607h25m34.816064247s) [upload=532346]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 22:54:34.944985461 (607h25m34.816068496s) [upload=532346]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 22:54:35.242687223 (607h25m35.113770258s) [upload=532346]: Done uploading segments... Dec 30 22:54:35.242705969 (607h25m35.113789014s) [upload=532346]: Upload finished. err: <nil> Duration: 9.151046088s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)