Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X....] duration: 26.787450925s trace: https://jaeger.dev.storj.io/trace/59e578b990080441 FAIL (2025-12-21 12:50:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ) context deadline exceeded ======LOGS====== Dec 21 12:47:56.652172038 (381h18m56.523255083s) [upload=334635]: Starting upload Dec 21 12:47:56.652261698 (381h18m56.523344743s) [upload=334635]: Done starting upload Dec 21 12:47:56.652383059 (381h18m56.523466105s) [upload=334635]: Uploading segments... Dec 21 12:47:56.652621874 (381h18m56.523704919s) [upload=334635]: Got next segment. Inline: false Dec 21 12:47:56.652662482 (381h18m56.523745547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:47:56.652668312 (381h18m56.523751358s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:47:58.310054254 (381h18m58.181137289s) [seg_pos={0 0}, upload=334635]: Begin upload segment... Dec 21 12:47:58.310100261 (381h18m58.181183296s) [seg_pos={0 0}, upload=334635]: Done begin upload segment. Dec 21 12:47:58.310125339 (381h18m58.181208385s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310200472 (381h18m58.181283507s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310860850 (381h18m58.181943885s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.311476922 (381h18m58.182559977s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312084460 (381h18m58.183167505s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312545497 (381h18m58.183628542s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312992288 (381h18m58.184075333s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314048429 (381h18m58.185131464s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314367226 (381h18m58.185450291s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315403740 (381h18m58.186486775s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315518408 (381h18m58.186601453s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.316594547 (381h18m58.187677582s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.317683261 (381h18m58.188766296s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319247959 (381h18m58.190331005s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319350675 (381h18m58.19043372s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.320362061 (381h18m58.191445106s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:48:00.166294046 (381h19m0.037377091s) [upload=334635]: Next returned nil segment Dec 21 12:48:00.166310136 (381h19m0.037393171s) [upload=334635]: Waiting for error group managing segments... Dec 21 12:48:00.490446417 (381h19m0.361529462s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.518873798 (381h19m0.389956843s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.570146999 (381h19m0.441230034s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.575795434 (381h19m0.446878479s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.629546113 (381h19m0.500629158s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.729323594 (381h19m0.600406639s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.783374374 (381h19m0.654457419s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.906375847 (381h19m0.777458892s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.211961276 (381h19m1.083044321s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.329628992 (381h19m1.200712047s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.364936753 (381h19m1.236019798s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.473683763 (381h19m1.344766818s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882295018 (381h19m1.753378063s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882423833 (381h19m1.753506878s) [seg_pos={0 0}, upload=334635]: Segment reached optimal threshold of 13 pieces. Dec 21 12:48:01.882682856 (381h19m1.753765901s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.882693497 (381h19m1.753776542s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883117604 (381h19m1.754200649s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883125709 (381h19m1.754208755s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883288981 (381h19m1.754372026s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883296104 (381h19m1.754379149s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883315130 (381h19m1.754398175s) [seg_pos={0 0}, upload=334635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:48:01.883399341 (381h19m1.754482386s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:48:01.883405502 (381h19m1.754488547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:48:02.189523284 (381h19m2.060606329s) [upload=334635]: Done uploading segments... Dec 21 12:48:02.189545897 (381h19m2.060628942s) [upload=334635]: Upload finished. err: <nil> Duration: 26.787480522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X...] duration: 5.707862973s trace: https://jaeger.dev.storj.io/trace/59e578b990080441 FAIL (2025-12-21 12:50:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ) context deadline exceeded ======LOGS====== Dec 21 12:47:56.652172038 (381h18m56.523255083s) [upload=334635]: Starting upload Dec 21 12:47:56.652261698 (381h18m56.523344743s) [upload=334635]: Done starting upload Dec 21 12:47:56.652383059 (381h18m56.523466105s) [upload=334635]: Uploading segments... Dec 21 12:47:56.652621874 (381h18m56.523704919s) [upload=334635]: Got next segment. Inline: false Dec 21 12:47:56.652662482 (381h18m56.523745547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:47:56.652668312 (381h18m56.523751358s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:47:58.310054254 (381h18m58.181137289s) [seg_pos={0 0}, upload=334635]: Begin upload segment... Dec 21 12:47:58.310100261 (381h18m58.181183296s) [seg_pos={0 0}, upload=334635]: Done begin upload segment. Dec 21 12:47:58.310125339 (381h18m58.181208385s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310200472 (381h18m58.181283507s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310860850 (381h18m58.181943885s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.311476922 (381h18m58.182559977s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312084460 (381h18m58.183167505s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312545497 (381h18m58.183628542s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312992288 (381h18m58.184075333s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314048429 (381h18m58.185131464s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314367226 (381h18m58.185450291s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315403740 (381h18m58.186486775s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315518408 (381h18m58.186601453s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.316594547 (381h18m58.187677582s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.317683261 (381h18m58.188766296s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319247959 (381h18m58.190331005s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319350675 (381h18m58.19043372s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.320362061 (381h18m58.191445106s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:48:00.166294046 (381h19m0.037377091s) [upload=334635]: Next returned nil segment Dec 21 12:48:00.166310136 (381h19m0.037393171s) [upload=334635]: Waiting for error group managing segments... Dec 21 12:48:00.490446417 (381h19m0.361529462s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.518873798 (381h19m0.389956843s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.570146999 (381h19m0.441230034s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.575795434 (381h19m0.446878479s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.629546113 (381h19m0.500629158s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.729323594 (381h19m0.600406639s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.783374374 (381h19m0.654457419s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.906375847 (381h19m0.777458892s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.211961276 (381h19m1.083044321s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.329628992 (381h19m1.200712047s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.364936753 (381h19m1.236019798s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.473683763 (381h19m1.344766818s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882295018 (381h19m1.753378063s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882423833 (381h19m1.753506878s) [seg_pos={0 0}, upload=334635]: Segment reached optimal threshold of 13 pieces. Dec 21 12:48:01.882682856 (381h19m1.753765901s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.882693497 (381h19m1.753776542s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883117604 (381h19m1.754200649s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883125709 (381h19m1.754208755s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883288981 (381h19m1.754372026s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883296104 (381h19m1.754379149s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883315130 (381h19m1.754398175s) [seg_pos={0 0}, upload=334635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:48:01.883399341 (381h19m1.754482386s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:48:01.883405502 (381h19m1.754488547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:48:02.189523284 (381h19m2.060606329s) [upload=334635]: Done uploading segments... Dec 21 12:48:02.189545897 (381h19m2.060628942s) [upload=334635]: Upload finished. err: <nil> Duration: 5.70789851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X..] duration: 6.016257362s trace: https://jaeger.dev.storj.io/trace/59e578b990080441 FAIL (2025-12-21 12:49:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ) context deadline exceeded ======LOGS====== Dec 21 12:47:56.652172038 (381h18m56.523255083s) [upload=334635]: Starting upload Dec 21 12:47:56.652261698 (381h18m56.523344743s) [upload=334635]: Done starting upload Dec 21 12:47:56.652383059 (381h18m56.523466105s) [upload=334635]: Uploading segments... Dec 21 12:47:56.652621874 (381h18m56.523704919s) [upload=334635]: Got next segment. Inline: false Dec 21 12:47:56.652662482 (381h18m56.523745547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:47:56.652668312 (381h18m56.523751358s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:47:58.310054254 (381h18m58.181137289s) [seg_pos={0 0}, upload=334635]: Begin upload segment... Dec 21 12:47:58.310100261 (381h18m58.181183296s) [seg_pos={0 0}, upload=334635]: Done begin upload segment. Dec 21 12:47:58.310125339 (381h18m58.181208385s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310200472 (381h18m58.181283507s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310860850 (381h18m58.181943885s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.311476922 (381h18m58.182559977s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312084460 (381h18m58.183167505s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312545497 (381h18m58.183628542s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312992288 (381h18m58.184075333s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314048429 (381h18m58.185131464s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314367226 (381h18m58.185450291s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315403740 (381h18m58.186486775s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315518408 (381h18m58.186601453s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.316594547 (381h18m58.187677582s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.317683261 (381h18m58.188766296s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319247959 (381h18m58.190331005s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319350675 (381h18m58.19043372s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.320362061 (381h18m58.191445106s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:48:00.166294046 (381h19m0.037377091s) [upload=334635]: Next returned nil segment Dec 21 12:48:00.166310136 (381h19m0.037393171s) [upload=334635]: Waiting for error group managing segments... Dec 21 12:48:00.490446417 (381h19m0.361529462s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.518873798 (381h19m0.389956843s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.570146999 (381h19m0.441230034s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.575795434 (381h19m0.446878479s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.629546113 (381h19m0.500629158s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.729323594 (381h19m0.600406639s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.783374374 (381h19m0.654457419s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.906375847 (381h19m0.777458892s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.211961276 (381h19m1.083044321s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.329628992 (381h19m1.200712047s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.364936753 (381h19m1.236019798s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.473683763 (381h19m1.344766818s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882295018 (381h19m1.753378063s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882423833 (381h19m1.753506878s) [seg_pos={0 0}, upload=334635]: Segment reached optimal threshold of 13 pieces. Dec 21 12:48:01.882682856 (381h19m1.753765901s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.882693497 (381h19m1.753776542s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883117604 (381h19m1.754200649s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883125709 (381h19m1.754208755s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883288981 (381h19m1.754372026s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883296104 (381h19m1.754379149s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883315130 (381h19m1.754398175s) [seg_pos={0 0}, upload=334635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:48:01.883399341 (381h19m1.754482386s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:48:01.883405502 (381h19m1.754488547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:48:02.189523284 (381h19m2.060606329s) [upload=334635]: Done uploading segments... Dec 21 12:48:02.189545897 (381h19m2.060628942s) [upload=334635]: Upload finished. err: <nil> Duration: 6.016290945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X.] duration: 6.019274205s trace: https://jaeger.dev.storj.io/trace/59e578b990080441 FAIL (2025-12-21 12:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ) context deadline exceeded ======LOGS====== Dec 21 12:47:56.652172038 (381h18m56.523255083s) [upload=334635]: Starting upload Dec 21 12:47:56.652261698 (381h18m56.523344743s) [upload=334635]: Done starting upload Dec 21 12:47:56.652383059 (381h18m56.523466105s) [upload=334635]: Uploading segments... Dec 21 12:47:56.652621874 (381h18m56.523704919s) [upload=334635]: Got next segment. Inline: false Dec 21 12:47:56.652662482 (381h18m56.523745547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:47:56.652668312 (381h18m56.523751358s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:47:58.310054254 (381h18m58.181137289s) [seg_pos={0 0}, upload=334635]: Begin upload segment... Dec 21 12:47:58.310100261 (381h18m58.181183296s) [seg_pos={0 0}, upload=334635]: Done begin upload segment. Dec 21 12:47:58.310125339 (381h18m58.181208385s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310200472 (381h18m58.181283507s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310860850 (381h18m58.181943885s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.311476922 (381h18m58.182559977s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312084460 (381h18m58.183167505s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312545497 (381h18m58.183628542s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312992288 (381h18m58.184075333s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314048429 (381h18m58.185131464s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314367226 (381h18m58.185450291s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315403740 (381h18m58.186486775s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315518408 (381h18m58.186601453s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.316594547 (381h18m58.187677582s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.317683261 (381h18m58.188766296s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319247959 (381h18m58.190331005s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319350675 (381h18m58.19043372s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.320362061 (381h18m58.191445106s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:48:00.166294046 (381h19m0.037377091s) [upload=334635]: Next returned nil segment Dec 21 12:48:00.166310136 (381h19m0.037393171s) [upload=334635]: Waiting for error group managing segments... Dec 21 12:48:00.490446417 (381h19m0.361529462s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.518873798 (381h19m0.389956843s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.570146999 (381h19m0.441230034s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.575795434 (381h19m0.446878479s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.629546113 (381h19m0.500629158s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.729323594 (381h19m0.600406639s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.783374374 (381h19m0.654457419s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.906375847 (381h19m0.777458892s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.211961276 (381h19m1.083044321s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.329628992 (381h19m1.200712047s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.364936753 (381h19m1.236019798s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.473683763 (381h19m1.344766818s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882295018 (381h19m1.753378063s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882423833 (381h19m1.753506878s) [seg_pos={0 0}, upload=334635]: Segment reached optimal threshold of 13 pieces. Dec 21 12:48:01.882682856 (381h19m1.753765901s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.882693497 (381h19m1.753776542s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883117604 (381h19m1.754200649s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883125709 (381h19m1.754208755s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883288981 (381h19m1.754372026s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883296104 (381h19m1.754379149s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883315130 (381h19m1.754398175s) [seg_pos={0 0}, upload=334635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:48:01.883399341 (381h19m1.754482386s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:48:01.883405502 (381h19m1.754488547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:48:02.189523284 (381h19m2.060606329s) [upload=334635]: Done uploading segments... Dec 21 12:48:02.189545897 (381h19m2.060628942s) [upload=334635]: Upload finished. err: <nil> Duration: 6.019302129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X] duration: 30.002238281s trace: https://jaeger.dev.storj.io/trace/59e578b990080441 FAIL (2025-12-21 12:48:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A) context deadline exceeded; piecestore: (Node ID: 12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, Piece ID: X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: 37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ) context deadline exceeded ======LOGS====== Dec 21 12:47:56.652172038 (381h18m56.523255083s) [upload=334635]: Starting upload Dec 21 12:47:56.652261698 (381h18m56.523344743s) [upload=334635]: Done starting upload Dec 21 12:47:56.652383059 (381h18m56.523466105s) [upload=334635]: Uploading segments... Dec 21 12:47:56.652621874 (381h18m56.523704919s) [upload=334635]: Got next segment. Inline: false Dec 21 12:47:56.652662482 (381h18m56.523745547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:47:56.652668312 (381h18m56.523751358s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:47:58.310054254 (381h18m58.181137289s) [seg_pos={0 0}, upload=334635]: Begin upload segment... Dec 21 12:47:58.310100261 (381h18m58.181183296s) [seg_pos={0 0}, upload=334635]: Done begin upload segment. Dec 21 12:47:58.310125339 (381h18m58.181208385s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310200472 (381h18m58.181283507s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.310860850 (381h18m58.181943885s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.311476922 (381h18m58.182559977s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312084460 (381h18m58.183167505s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312545497 (381h18m58.183628542s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.312992288 (381h18m58.184075333s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314048429 (381h18m58.185131464s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.314367226 (381h18m58.185450291s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315403740 (381h18m58.186486775s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.315518408 (381h18m58.186601453s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.316594547 (381h18m58.187677582s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.317683261 (381h18m58.188766296s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319247959 (381h18m58.190331005s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.319350675 (381h18m58.19043372s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:47:58.320362061 (381h18m58.191445106s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Uploading piece... Dec 21 12:48:00.166294046 (381h19m0.037377091s) [upload=334635]: Next returned nil segment Dec 21 12:48:00.166310136 (381h19m0.037393171s) [upload=334635]: Waiting for error group managing segments... Dec 21 12:48:00.490446417 (381h19m0.361529462s) [noise=true , address=145.79.227.43:4157 , piece_id=5YVET2RXPY3253P6XUU4XG2QWZX5D4IVPJI27RNUR7Y4JUW2OYEQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.518873798 (381h19m0.389956843s) [noise=true , address=145.79.227.52:4446 , piece_id=LBCEZSDRHVOPUGRPZZW6SYZDKESSYXAE7O7UTQKXF7LUIHY2OTMQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.570146999 (381h19m0.441230034s) [noise=true , address=45.91.116.8:4046 , piece_id=ZQEDW3SJG2HX55SE4UZA5L6OSX4Z62ND7R7JM6DEW3RLLFOKKL3A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.575795434 (381h19m0.446878479s) [noise=true , address=145.79.227.48:4303 , piece_id=X3HRBTDWPSKL27LXY5WYM2WXCXLBEIWIHQQ5BX4PAE5UWVIQJFRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.629546113 (381h19m0.500629158s) [noise=true , address=45.91.116.14:4192 , piece_id=736HI52H7PQSCNYO6QGEXA4OMFU7464IHOQVMJT2FF7CYNLILHJQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.729323594 (381h19m0.600406639s) [noise=true , address=145.79.227.34:4906 , piece_id=SJ5P5XDOXYCIQTCC4QNSD3W3MFX55W3RV2K5YQIIVTANEK46SHRQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.783374374 (381h19m0.654457419s) [noise=true , address=145.79.227.9:4366 , piece_id=QO3G5YSVQSBCMEMB4RN6X5GCJJRXHWEKFBKCDGJ3PA4ASJTXNL6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:00.906375847 (381h19m0.777458892s) [noise=true , address=145.79.227.15:4875 , piece_id=64SZZMKS63Y6NOWHL3ZXY34ZDI7PNP3C3GHW5MBZIRQ7MJVQHGQQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.211961276 (381h19m1.083044321s) [noise=true , address=145.79.227.39:4015 , piece_id=37P3YJQZWIMWN2HHZWGYKX5J7PBYJDESIS3H3HYFJBTGU7ACARYA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.329628992 (381h19m1.200712047s) [noise=true , address=77.90.31.21:4880 , piece_id=KMANQWXOMKSO3I6VSBQFINO54T67AN6C4FPYG7DKCIPYPUUBMTWA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.364936753 (381h19m1.236019798s) [noise=true , address=45.91.116.9:4264 , piece_id=F6LQEIO4TP6FPTBH566V6IQGBC6WDBMKDVNCOQVBRHFETAUPDWUQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.473683763 (381h19m1.344766818s) [noise=true , address=145.79.227.32:4834 , piece_id=YFDD7MI4SVYANSQNGHEPEE4KKBZDL5WMV6ZNZSCSDJYA7B6WJE6A, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882295018 (381h19m1.753378063s) [noise=true , address=145.79.227.51:4409 , piece_id=LRT7BWWKY6LLOWDXJNIWOALQIAFD52LRHCOAFRV6DCMWJJPRZ6XQ, seg_pos={0 0}, upload=334635]: Done uploading piece. err: <nil> Dec 21 12:48:01.882423833 (381h19m1.753506878s) [seg_pos={0 0}, upload=334635]: Segment reached optimal threshold of 13 pieces. Dec 21 12:48:01.882682856 (381h19m1.753765901s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.882693497 (381h19m1.753776542s) [noise=true , address=145.79.227.50:4374 , piece_id=COCAWDF3DIT7UDAJIUKXFGGVRXSCKISDWRWYXEIZENA3LU5MBKUA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883117604 (381h19m1.754200649s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883125709 (381h19m1.754208755s) [noise=true , address=145.79.227.44:4195 , piece_id=C36VVKNHZZAO6P3BSDUFRMC5BTCEYU2XTIAZHFC76ZVWQ3F5KOBA, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883288981 (381h19m1.754372026s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 21 12:48:01.883296104 (381h19m1.754379149s) [noise=true , address=145.79.227.30:4761 , piece_id=O72ZQE55CGBN4OXUWQQ64CNKEFXOOYTNLF64JE2RTL4PLJUXYB4Q, seg_pos={0 0}, upload=334635]: Long tail cancellation due to optimal threshold Dec 21 12:48:01.883315130 (381h19m1.754398175s) [seg_pos={0 0}, upload=334635]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:48:01.883399341 (381h19m1.754482386s) [upload=334635]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:48:01.883405502 (381h19m1.754488547s) [upload=334635]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:48:02.189523284 (381h19m2.060606329s) [upload=334635]: Done uploading segments... Dec 21 12:48:02.189545897 (381h19m2.060628942s) [upload=334635]: Upload finished. err: <nil> Duration: 30.002256216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X....] duration: 5.700834287s trace: https://jaeger.dev.storj.io/trace/1f7568dcc7607df1 FAIL (2025-12-21 12:46:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: 2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ) context deadline exceeded ======LOGS====== 12:43:23.138370507 (381h14m23.009453542s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:43:23.138376068 (381h14m23.009459113s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:43:23.402588707 (381h14m23.273671742s) [seg_pos={0 0}, upload=334568]: Begin upload segment... Dec 21 12:43:23.402634994 (381h14m23.273718039s) [seg_pos={0 0}, upload=334568]: Done begin upload segment. Dec 21 12:43:23.402660924 (381h14m23.273743959s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.402790591 (381h14m23.273873626s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403254474 (381h14m23.274337519s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403568071 (381h14m23.274651106s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403853895 (381h14m23.27493693s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404210524 (381h14m23.275293559s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404611809 (381h14m23.275694844s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404905508 (381h14m23.275988553s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.405225196 (381h14m23.276308231s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406637586 (381h14m23.277720621s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406849790 (381h14m23.277932835s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.408168271 (381h14m23.279251306s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.409391279 (381h14m23.280474315s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.411101686 (381h14m23.282184721s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.412912144 (381h14m23.283995179s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.414510698 (381h14m23.285593733s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:25.250954861 (381h14m25.122037906s) [upload=334568]: Next returned nil segment Dec 21 12:43:25.250971653 (381h14m25.122054688s) [upload=334568]: Waiting for error group managing segments... Dec 21 12:43:25.557842890 (381h14m25.428925925s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.598855795 (381h14m25.46993884s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.615172283 (381h14m25.486255329s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.650771018 (381h14m25.521854053s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.658466341 (381h14m25.529549376s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.686542312 (381h14m25.557625347s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.724680333 (381h14m25.595763378s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.785567290 (381h14m25.656650325s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.849584839 (381h14m25.720667894s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.314502658 (381h14m26.185585693s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.365678814 (381h14m26.236761859s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.428161740 (381h14m26.299244775s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578834252 (381h14m26.449917287s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578852488 (381h14m26.449935533s) [seg_pos={0 0}, upload=334568]: Segment reached optimal threshold of 13 pieces. Dec 21 12:43:26.579112362 (381h14m26.450195407s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection Dec 21 12:43:26.579122611 (381h14m26.450205656s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579132100 (381h14m26.450215135s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: read tcp 188.166.225.170:60836->145.79.227.36:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:43:26.579141988 (381h14m26.450225024s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579250485 (381h14m26.45033352s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection Dec 21 12:43:26.579255795 (381h14m26.45033883s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579269782 (381h14m26.450352827s) [seg_pos={0 0}, upload=334568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:43:26.579356497 (381h14m26.450439532s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:43:26.579361566 (381h14m26.450444601s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:43:29.595876446 (381h14m29.466959481s) [upload=334568]: Done uploading segments... Dec 21 12:43:29.595895932 (381h14m29.466978967s) [upload=334568]: Upload finished. err: <nil> Duration: 5.700866026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X...] duration: 5.879702665s trace: https://jaeger.dev.storj.io/trace/1f7568dcc7607df1 FAIL (2025-12-21 12:45:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: 2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ) context deadline exceeded ======LOGS====== 12:43:23.138370507 (381h14m23.009453542s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:43:23.138376068 (381h14m23.009459113s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:43:23.402588707 (381h14m23.273671742s) [seg_pos={0 0}, upload=334568]: Begin upload segment... Dec 21 12:43:23.402634994 (381h14m23.273718039s) [seg_pos={0 0}, upload=334568]: Done begin upload segment. Dec 21 12:43:23.402660924 (381h14m23.273743959s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.402790591 (381h14m23.273873626s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403254474 (381h14m23.274337519s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403568071 (381h14m23.274651106s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403853895 (381h14m23.27493693s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404210524 (381h14m23.275293559s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404611809 (381h14m23.275694844s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404905508 (381h14m23.275988553s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.405225196 (381h14m23.276308231s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406637586 (381h14m23.277720621s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406849790 (381h14m23.277932835s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.408168271 (381h14m23.279251306s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.409391279 (381h14m23.280474315s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.411101686 (381h14m23.282184721s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.412912144 (381h14m23.283995179s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.414510698 (381h14m23.285593733s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:25.250954861 (381h14m25.122037906s) [upload=334568]: Next returned nil segment Dec 21 12:43:25.250971653 (381h14m25.122054688s) [upload=334568]: Waiting for error group managing segments... Dec 21 12:43:25.557842890 (381h14m25.428925925s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.598855795 (381h14m25.46993884s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.615172283 (381h14m25.486255329s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.650771018 (381h14m25.521854053s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.658466341 (381h14m25.529549376s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.686542312 (381h14m25.557625347s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.724680333 (381h14m25.595763378s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.785567290 (381h14m25.656650325s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.849584839 (381h14m25.720667894s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.314502658 (381h14m26.185585693s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.365678814 (381h14m26.236761859s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.428161740 (381h14m26.299244775s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578834252 (381h14m26.449917287s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578852488 (381h14m26.449935533s) [seg_pos={0 0}, upload=334568]: Segment reached optimal threshold of 13 pieces. Dec 21 12:43:26.579112362 (381h14m26.450195407s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection Dec 21 12:43:26.579122611 (381h14m26.450205656s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579132100 (381h14m26.450215135s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: read tcp 188.166.225.170:60836->145.79.227.36:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:43:26.579141988 (381h14m26.450225024s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579250485 (381h14m26.45033352s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection Dec 21 12:43:26.579255795 (381h14m26.45033883s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579269782 (381h14m26.450352827s) [seg_pos={0 0}, upload=334568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:43:26.579356497 (381h14m26.450439532s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:43:26.579361566 (381h14m26.450444601s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:43:29.595876446 (381h14m29.466959481s) [upload=334568]: Done uploading segments... Dec 21 12:43:29.595895932 (381h14m29.466978967s) [upload=334568]: Upload finished. err: <nil> Duration: 5.879717735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X..] duration: 5.733727569s trace: https://jaeger.dev.storj.io/trace/1f7568dcc7607df1 FAIL (2025-12-21 12:44:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: 2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ) context deadline exceeded ======LOGS====== 12:43:23.138370507 (381h14m23.009453542s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:43:23.138376068 (381h14m23.009459113s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:43:23.402588707 (381h14m23.273671742s) [seg_pos={0 0}, upload=334568]: Begin upload segment... Dec 21 12:43:23.402634994 (381h14m23.273718039s) [seg_pos={0 0}, upload=334568]: Done begin upload segment. Dec 21 12:43:23.402660924 (381h14m23.273743959s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.402790591 (381h14m23.273873626s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403254474 (381h14m23.274337519s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403568071 (381h14m23.274651106s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403853895 (381h14m23.27493693s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404210524 (381h14m23.275293559s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404611809 (381h14m23.275694844s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404905508 (381h14m23.275988553s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.405225196 (381h14m23.276308231s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406637586 (381h14m23.277720621s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406849790 (381h14m23.277932835s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.408168271 (381h14m23.279251306s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.409391279 (381h14m23.280474315s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.411101686 (381h14m23.282184721s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.412912144 (381h14m23.283995179s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.414510698 (381h14m23.285593733s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:25.250954861 (381h14m25.122037906s) [upload=334568]: Next returned nil segment Dec 21 12:43:25.250971653 (381h14m25.122054688s) [upload=334568]: Waiting for error group managing segments... Dec 21 12:43:25.557842890 (381h14m25.428925925s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.598855795 (381h14m25.46993884s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.615172283 (381h14m25.486255329s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.650771018 (381h14m25.521854053s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.658466341 (381h14m25.529549376s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.686542312 (381h14m25.557625347s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.724680333 (381h14m25.595763378s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.785567290 (381h14m25.656650325s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.849584839 (381h14m25.720667894s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.314502658 (381h14m26.185585693s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.365678814 (381h14m26.236761859s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.428161740 (381h14m26.299244775s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578834252 (381h14m26.449917287s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578852488 (381h14m26.449935533s) [seg_pos={0 0}, upload=334568]: Segment reached optimal threshold of 13 pieces. Dec 21 12:43:26.579112362 (381h14m26.450195407s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection Dec 21 12:43:26.579122611 (381h14m26.450205656s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579132100 (381h14m26.450215135s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: read tcp 188.166.225.170:60836->145.79.227.36:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:43:26.579141988 (381h14m26.450225024s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579250485 (381h14m26.45033352s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection Dec 21 12:43:26.579255795 (381h14m26.45033883s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579269782 (381h14m26.450352827s) [seg_pos={0 0}, upload=334568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:43:26.579356497 (381h14m26.450439532s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:43:26.579361566 (381h14m26.450444601s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:43:29.595876446 (381h14m29.466959481s) [upload=334568]: Done uploading segments... Dec 21 12:43:29.595895932 (381h14m29.466978967s) [upload=334568]: Upload finished. err: <nil> Duration: 5.733755543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X.] duration: 5.920501905s trace: https://jaeger.dev.storj.io/trace/1f7568dcc7607df1 FAIL (2025-12-21 12:44:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: 2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ) context deadline exceeded ======LOGS====== 12:43:23.138370507 (381h14m23.009453542s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:43:23.138376068 (381h14m23.009459113s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:43:23.402588707 (381h14m23.273671742s) [seg_pos={0 0}, upload=334568]: Begin upload segment... Dec 21 12:43:23.402634994 (381h14m23.273718039s) [seg_pos={0 0}, upload=334568]: Done begin upload segment. Dec 21 12:43:23.402660924 (381h14m23.273743959s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.402790591 (381h14m23.273873626s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403254474 (381h14m23.274337519s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403568071 (381h14m23.274651106s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403853895 (381h14m23.27493693s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404210524 (381h14m23.275293559s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404611809 (381h14m23.275694844s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404905508 (381h14m23.275988553s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.405225196 (381h14m23.276308231s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406637586 (381h14m23.277720621s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406849790 (381h14m23.277932835s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.408168271 (381h14m23.279251306s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.409391279 (381h14m23.280474315s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.411101686 (381h14m23.282184721s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.412912144 (381h14m23.283995179s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.414510698 (381h14m23.285593733s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:25.250954861 (381h14m25.122037906s) [upload=334568]: Next returned nil segment Dec 21 12:43:25.250971653 (381h14m25.122054688s) [upload=334568]: Waiting for error group managing segments... Dec 21 12:43:25.557842890 (381h14m25.428925925s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.598855795 (381h14m25.46993884s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.615172283 (381h14m25.486255329s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.650771018 (381h14m25.521854053s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.658466341 (381h14m25.529549376s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.686542312 (381h14m25.557625347s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.724680333 (381h14m25.595763378s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.785567290 (381h14m25.656650325s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.849584839 (381h14m25.720667894s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.314502658 (381h14m26.185585693s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.365678814 (381h14m26.236761859s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.428161740 (381h14m26.299244775s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578834252 (381h14m26.449917287s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578852488 (381h14m26.449935533s) [seg_pos={0 0}, upload=334568]: Segment reached optimal threshold of 13 pieces. Dec 21 12:43:26.579112362 (381h14m26.450195407s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection Dec 21 12:43:26.579122611 (381h14m26.450205656s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579132100 (381h14m26.450215135s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: read tcp 188.166.225.170:60836->145.79.227.36:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:43:26.579141988 (381h14m26.450225024s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579250485 (381h14m26.45033352s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection Dec 21 12:43:26.579255795 (381h14m26.45033883s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579269782 (381h14m26.450352827s) [seg_pos={0 0}, upload=334568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:43:26.579356497 (381h14m26.450439532s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:43:26.579361566 (381h14m26.450444601s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:43:29.595876446 (381h14m29.466959481s) [upload=334568]: Done uploading segments... Dec 21 12:43:29.595895932 (381h14m29.466978967s) [upload=334568]: Upload finished. err: <nil> Duration: 5.920531161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..X] duration: 30.002332499s trace: https://jaeger.dev.storj.io/trace/1f7568dcc7607df1 FAIL (2025-12-21 12:43:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, Piece ID: L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA) context deadline exceeded; piecestore: (Node ID: 1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, Piece ID: 2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: 6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ) context deadline exceeded; piecestore: (Node ID: 12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, Piece ID: KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ) context deadline exceeded ======LOGS====== 12:43:23.138370507 (381h14m23.009453542s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:43:23.138376068 (381h14m23.009459113s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:43:23.402588707 (381h14m23.273671742s) [seg_pos={0 0}, upload=334568]: Begin upload segment... Dec 21 12:43:23.402634994 (381h14m23.273718039s) [seg_pos={0 0}, upload=334568]: Done begin upload segment. Dec 21 12:43:23.402660924 (381h14m23.273743959s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.402790591 (381h14m23.273873626s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403254474 (381h14m23.274337519s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403568071 (381h14m23.274651106s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.403853895 (381h14m23.27493693s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404210524 (381h14m23.275293559s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404611809 (381h14m23.275694844s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.404905508 (381h14m23.275988553s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.405225196 (381h14m23.276308231s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406637586 (381h14m23.277720621s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.406849790 (381h14m23.277932835s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.408168271 (381h14m23.279251306s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.409391279 (381h14m23.280474315s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.411101686 (381h14m23.282184721s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.412912144 (381h14m23.283995179s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:23.414510698 (381h14m23.285593733s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Uploading piece... Dec 21 12:43:25.250954861 (381h14m25.122037906s) [upload=334568]: Next returned nil segment Dec 21 12:43:25.250971653 (381h14m25.122054688s) [upload=334568]: Waiting for error group managing segments... Dec 21 12:43:25.557842890 (381h14m25.428925925s) [noise=true , address=145.79.227.20:4439 , piece_id=2E3SJN7IASNF74OOAIVYVDO2QC74EWSAN6EQ2KHVBNIICNJNP6WQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.598855795 (381h14m25.46993884s) [noise=true , address=145.79.227.43:4157 , piece_id=6JM27THZJFQRYO6K7EJXTESXRNJXSVBI4M2QDRTKGATZ5Z5N7N3A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.615172283 (381h14m25.486255329s) [noise=true , address=145.79.227.4:4186 , piece_id=LJ2G6WKLQNMFYQWUXK3Y5HV65TJ26JBSEI2B2GW3FSH6SHWFM7HQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.650771018 (381h14m25.521854053s) [noise=true , address=145.79.227.32:4835 , piece_id=L44PCCEUU4F5HHTPAZY2ACP5QQ7JPGXT4KYFWTJYGV7WRT5D47CA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.658466341 (381h14m25.529549376s) [noise=true , address=145.79.227.52:4445 , piece_id=AXL3D2SFYKOXJIMQUDNYPZYYW4IATTA7ZXM3LBOYSQPJ4B4C2JNA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.686542312 (381h14m25.557625347s) [noise=true , address=145.79.227.29:4942 , piece_id=6TYH4GZBFDQFFWBJYZ6N4YGTLEH2IRYHHWDVKZOQE5PZNDFZ2OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.724680333 (381h14m25.595763378s) [noise=true , address=45.91.116.4:4937 , piece_id=BKVCDMJHCSRIDX2XWTD72UFM4CPOUPRVN5SKAGNPVRBCJZYQ5VZA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.785567290 (381h14m25.656650325s) [noise=true , address=77.90.31.11:4555 , piece_id=5MEMASI37PSF7L6G3Q3UUMZEAWJ33Q4GFH2AJTODSZFMNQR3EKAQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:25.849584839 (381h14m25.720667894s) [noise=true , address=45.91.116.9:4261 , piece_id=RFFOPQ3M2EPRR6HLBIZWMSXF7U7B45WVJEJU547QTCNPAXUX4QGA, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.314502658 (381h14m26.185585693s) [noise=true , address=145.79.227.42:4122 , piece_id=RUA7FQ2IXOMQLSSX5IT64CAWKNDO2SXZFXNECAPYG273Z3JL5P2Q, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.365678814 (381h14m26.236761859s) [noise=true , address=145.79.227.8:4330 , piece_id=BWRHXKHULSRMKXRJONUWQDQB5FUBEJJ2DYWUOB3HECFQR7UHELXQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.428161740 (381h14m26.299244775s) [noise=true , address=77.90.31.13:4629 , piece_id=TAAVP3SUSH4IQLYFCPA5PK3BV54XPZV2EIRIAWQTW27L77VHQDKQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578834252 (381h14m26.449917287s) [noise=true , address=145.79.227.10:4403 , piece_id=KTEINXC6X7NVYQ6N6QOEB2IWG3TOPX6ZYQEZI2UPWRW6JVOB2JBQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: <nil> Dec 21 12:43:26.578852488 (381h14m26.449935533s) [seg_pos={0 0}, upload=334568]: Segment reached optimal threshold of 13 pieces. Dec 21 12:43:26.579112362 (381h14m26.450195407s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47032->145.79.227.26:4653: use of closed network connection Dec 21 12:43:26.579122611 (381h14m26.450205656s) [noise=true , address=145.79.227.26:4653 , piece_id=V4VLA7Y5FYMCF47QPAEYG4FGW3DHZ6OZDXYMIITK33TFMNWDYDLQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579132100 (381h14m26.450215135s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb): protocol: read tcp 188.166.225.170:60836->145.79.227.36:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:43:26.579141988 (381h14m26.450225024s) [noise=true , address=145.79.227.36:4518 , piece_id=BPXOPHCW7K3CQM2C2GMYGP2NOKQOTEIJE4BI6WN73EBNL5N6NJ6A, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579250485 (381h14m26.45033352s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36778->145.79.227.37:4567: use of closed network connection Dec 21 12:43:26.579255795 (381h14m26.45033883s) [noise=true , address=145.79.227.37:4567 , piece_id=W6ZQKNJFEAJNGBH5LAFMKUHRN34AHTU4FQCRDJAIS7LMGIEV5OIQ, seg_pos={0 0}, upload=334568]: Long tail cancellation due to optimal threshold Dec 21 12:43:26.579269782 (381h14m26.450352827s) [seg_pos={0 0}, upload=334568]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:43:26.579356497 (381h14m26.450439532s) [upload=334568]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:43:26.579361566 (381h14m26.450444601s) [upload=334568]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:43:29.595876446 (381h14m29.466959481s) [upload=334568]: Done uploading segments... Dec 21 12:43:29.595895932 (381h14m29.466978967s) [upload=334568]: Upload finished. err: <nil> Duration: 30.002356455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X..] duration: 5.934091882s trace: https://jaeger.dev.storj.io/trace/3f772a8b8d28b331 FAIL (2025-12-21 12:42:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ) context deadline exceeded; piecestore: (Node ID: 124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, Piece ID: H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA) context deadline exceeded ======LOGS====== Dec 21 12:40:46.128080141 (381h11m45.999163186s) [upload=334532]: Starting upload Dec 21 12:40:46.128164041 (381h11m45.999247076s) [upload=334532]: Done starting upload Dec 21 12:40:46.128283829 (381h11m45.999366874s) [upload=334532]: Uploading segments... Dec 21 12:40:46.128447340 (381h11m45.999530385s) [upload=334532]: Got next segment. Inline: false Dec 21 12:40:46.128495151 (381h11m45.999578196s) [upload=334532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:40:46.128501985 (381h11m45.99958502s) [upload=334532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:40:46.371673158 (381h11m46.242756233s) [seg_pos={0 0}, upload=334532]: Begin upload segment... Dec 21 12:40:46.371738763 (381h11m46.242821818s) [seg_pos={0 0}, upload=334532]: Done begin upload segment. Dec 21 12:40:46.371779020 (381h11m46.242862085s) [noise=true , address=145.79.227.45:4230 , piece_id=Y7CIVZTY4J7WTF7DH25EGTQHMJHKEHCCHZMRDRCBQVWLL6PUSKEQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.371838442 (381h11m46.242921477s) [noise=true , address=45.91.116.14:4190 , piece_id=BTMYSIYFECGOBMLINMV4EGJBSDR7SBQTBYJPVDIF7IPRKV4ZBKJQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.372455948 (381h11m46.243538983s) [noise=true , address=145.79.227.47:4266 , piece_id=M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.372882260 (381h11m46.243965295s) [noise=true , address=145.79.227.51:4409 , piece_id=2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.373667365 (381h11m46.24475041s) [noise=true , address=45.91.116.11:4120 , piece_id=DZY76BJDMMIW32EG3Y5YXB62EBCININT6GOHJZICIYYXGFSQU6XA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.373687093 (381h11m46.244770138s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.374798799 (381h11m46.245881834s) [noise=true , address=45.91.116.10:4083 , piece_id=QMO5GLTIPCL6URQKEYGFHBVMYYJGXGKXUFKZLYUN6DOA6I2AZ5LA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.374958784 (381h11m46.246041829s) [noise=true , address=145.79.227.42:4122 , piece_id=H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.375881512 (381h11m46.246964567s) [noise=true , address=145.79.227.20:4437 , piece_id=FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.376081542 (381h11m46.247164587s) [noise=true , address=45.91.116.12:4226 , piece_id=CZDJLBP4LNXK6TZEF7QTTS2K2AI3YWL5GOOFFBZ4EDC6EDT5FKGA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.377087808 (381h11m46.248170843s) [noise=true , address=45.91.116.8:4045 , piece_id=H4LJDYLU7HQB2H5AVBOHUJFAE4C7ZJVMHWJWY3YRKDKXJJXRWSEA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.377286877 (381h11m46.248369912s) [noise=true , address=77.90.31.11:4555 , piece_id=NKSBSQ5PZAKZSTNT5UJ4XGZVOR4D7Y6VBNI3BJNNG5M4CWLSCQBQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.378029000 (381h11m46.249112045s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.378367575 (381h11m46.24945061s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.380863277 (381h11m46.251946322s) [noise=true , address=145.79.227.50:4375 , piece_id=U2ITQWUUV4B652CMHM7HSAZCZFOMLHQSRDXR2SWOQEDV4I36I2JQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.387072851 (381h11m46.258155897s) [noise=true , address=145.79.227.41:4085 , piece_id=QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:48.136762127 (381h11m48.007845162s) [upload=334532]: Next returned nil segment Dec 21 12:40:48.136777005 (381h11m48.00786004s) [upload=334532]: Waiting for error group managing segments... Dec 21 12:40:48.536674798 (381h11m48.407757843s) [noise=true , address=145.79.227.47:4266 , piece_id=M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.543204462 (381h11m48.414287507s) [noise=true , address=145.79.227.42:4122 , piece_id=H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.580163226 (381h11m48.451246261s) [noise=true , address=45.91.116.8:4045 , piece_id=H4LJDYLU7HQB2H5AVBOHUJFAE4C7ZJVMHWJWY3YRKDKXJJXRWSEA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.581079802 (381h11m48.452162837s) [noise=true , address=145.79.227.20:4437 , piece_id=FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.611779188 (381h11m48.482862223s) [noise=true , address=145.79.227.45:4230 , piece_id=Y7CIVZTY4J7WTF7DH25EGTQHMJHKEHCCHZMRDRCBQVWLL6PUSKEQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.639282628 (381h11m48.510365673s) [noise=true , address=145.79.227.41:4085 , piece_id=QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.666980178 (381h11m48.538063213s) [noise=true , address=45.91.116.14:4190 , piece_id=BTMYSIYFECGOBMLINMV4EGJBSDR7SBQTBYJPVDIF7IPRKV4ZBKJQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.667503785 (381h11m48.53858682s) [noise=true , address=45.91.116.11:4120 , piece_id=DZY76BJDMMIW32EG3Y5YXB62EBCININT6GOHJZICIYYXGFSQU6XA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.668450628 (381h11m48.539533673s) [noise=true , address=77.90.31.11:4555 , piece_id=NKSBSQ5PZAKZSTNT5UJ4XGZVOR4D7Y6VBNI3BJNNG5M4CWLSCQBQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.690686959 (381h11m48.561769994s) [noise=true , address=45.91.116.12:4226 , piece_id=CZDJLBP4LNXK6TZEF7QTTS2K2AI3YWL5GOOFFBZ4EDC6EDT5FKGA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.844780205 (381h11m48.71586324s) [noise=true , address=145.79.227.51:4409 , piece_id=2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.260685007 (381h11m49.131768042s) [noise=true , address=145.79.227.50:4375 , piece_id=U2ITQWUUV4B652CMHM7HSAZCZFOMLHQSRDXR2SWOQEDV4I36I2JQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.283553241 (381h11m49.154636276s) [noise=true , address=45.91.116.10:4083 , piece_id=QMO5GLTIPCL6URQKEYGFHBVMYYJGXGKXUFKZLYUN6DOA6I2AZ5LA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.283572278 (381h11m49.154655313s) [seg_pos={0 0}, upload=334532]: Segment reached optimal threshold of 13 pieces. Dec 21 12:40:49.283738254 (381h11m49.154821289s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.283744285 (381h11m49.15482732s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.283805843 (381h11m49.154888878s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.283820700 (381h11m49.154903735s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.284005161 (381h11m49.155088196s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.284011584 (381h11m49.155094619s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.284030430 (381h11m49.155113465s) [seg_pos={0 0}, upload=334532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:40:49.285004034 (381h11m49.156087069s) [upload=334532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:40:49.285034291 (381h11m49.156117337s) [upload=334532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:40:49.569439270 (381h11m49.440522305s) [upload=334532]: Done uploading segments... Dec 21 12:40:49.569454108 (381h11m49.440537153s) [upload=334532]: Upload finished. err: <nil> Duration: 5.934121358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X.] duration: 29.240468117s trace: https://jaeger.dev.storj.io/trace/3f772a8b8d28b331 FAIL (2025-12-21 12:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ) context deadline exceeded; piecestore: (Node ID: 124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, Piece ID: H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA) context deadline exceeded ======LOGS====== Dec 21 12:40:46.128080141 (381h11m45.999163186s) [upload=334532]: Starting upload Dec 21 12:40:46.128164041 (381h11m45.999247076s) [upload=334532]: Done starting upload Dec 21 12:40:46.128283829 (381h11m45.999366874s) [upload=334532]: Uploading segments... Dec 21 12:40:46.128447340 (381h11m45.999530385s) [upload=334532]: Got next segment. Inline: false Dec 21 12:40:46.128495151 (381h11m45.999578196s) [upload=334532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:40:46.128501985 (381h11m45.99958502s) [upload=334532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:40:46.371673158 (381h11m46.242756233s) [seg_pos={0 0}, upload=334532]: Begin upload segment... Dec 21 12:40:46.371738763 (381h11m46.242821818s) [seg_pos={0 0}, upload=334532]: Done begin upload segment. Dec 21 12:40:46.371779020 (381h11m46.242862085s) [noise=true , address=145.79.227.45:4230 , piece_id=Y7CIVZTY4J7WTF7DH25EGTQHMJHKEHCCHZMRDRCBQVWLL6PUSKEQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.371838442 (381h11m46.242921477s) [noise=true , address=45.91.116.14:4190 , piece_id=BTMYSIYFECGOBMLINMV4EGJBSDR7SBQTBYJPVDIF7IPRKV4ZBKJQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.372455948 (381h11m46.243538983s) [noise=true , address=145.79.227.47:4266 , piece_id=M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.372882260 (381h11m46.243965295s) [noise=true , address=145.79.227.51:4409 , piece_id=2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.373667365 (381h11m46.24475041s) [noise=true , address=45.91.116.11:4120 , piece_id=DZY76BJDMMIW32EG3Y5YXB62EBCININT6GOHJZICIYYXGFSQU6XA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.373687093 (381h11m46.244770138s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.374798799 (381h11m46.245881834s) [noise=true , address=45.91.116.10:4083 , piece_id=QMO5GLTIPCL6URQKEYGFHBVMYYJGXGKXUFKZLYUN6DOA6I2AZ5LA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.374958784 (381h11m46.246041829s) [noise=true , address=145.79.227.42:4122 , piece_id=H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.375881512 (381h11m46.246964567s) [noise=true , address=145.79.227.20:4437 , piece_id=FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.376081542 (381h11m46.247164587s) [noise=true , address=45.91.116.12:4226 , piece_id=CZDJLBP4LNXK6TZEF7QTTS2K2AI3YWL5GOOFFBZ4EDC6EDT5FKGA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.377087808 (381h11m46.248170843s) [noise=true , address=45.91.116.8:4045 , piece_id=H4LJDYLU7HQB2H5AVBOHUJFAE4C7ZJVMHWJWY3YRKDKXJJXRWSEA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.377286877 (381h11m46.248369912s) [noise=true , address=77.90.31.11:4555 , piece_id=NKSBSQ5PZAKZSTNT5UJ4XGZVOR4D7Y6VBNI3BJNNG5M4CWLSCQBQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.378029000 (381h11m46.249112045s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.378367575 (381h11m46.24945061s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.380863277 (381h11m46.251946322s) [noise=true , address=145.79.227.50:4375 , piece_id=U2ITQWUUV4B652CMHM7HSAZCZFOMLHQSRDXR2SWOQEDV4I36I2JQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.387072851 (381h11m46.258155897s) [noise=true , address=145.79.227.41:4085 , piece_id=QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:48.136762127 (381h11m48.007845162s) [upload=334532]: Next returned nil segment Dec 21 12:40:48.136777005 (381h11m48.00786004s) [upload=334532]: Waiting for error group managing segments... Dec 21 12:40:48.536674798 (381h11m48.407757843s) [noise=true , address=145.79.227.47:4266 , piece_id=M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.543204462 (381h11m48.414287507s) [noise=true , address=145.79.227.42:4122 , piece_id=H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.580163226 (381h11m48.451246261s) [noise=true , address=45.91.116.8:4045 , piece_id=H4LJDYLU7HQB2H5AVBOHUJFAE4C7ZJVMHWJWY3YRKDKXJJXRWSEA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.581079802 (381h11m48.452162837s) [noise=true , address=145.79.227.20:4437 , piece_id=FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.611779188 (381h11m48.482862223s) [noise=true , address=145.79.227.45:4230 , piece_id=Y7CIVZTY4J7WTF7DH25EGTQHMJHKEHCCHZMRDRCBQVWLL6PUSKEQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.639282628 (381h11m48.510365673s) [noise=true , address=145.79.227.41:4085 , piece_id=QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.666980178 (381h11m48.538063213s) [noise=true , address=45.91.116.14:4190 , piece_id=BTMYSIYFECGOBMLINMV4EGJBSDR7SBQTBYJPVDIF7IPRKV4ZBKJQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.667503785 (381h11m48.53858682s) [noise=true , address=45.91.116.11:4120 , piece_id=DZY76BJDMMIW32EG3Y5YXB62EBCININT6GOHJZICIYYXGFSQU6XA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.668450628 (381h11m48.539533673s) [noise=true , address=77.90.31.11:4555 , piece_id=NKSBSQ5PZAKZSTNT5UJ4XGZVOR4D7Y6VBNI3BJNNG5M4CWLSCQBQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.690686959 (381h11m48.561769994s) [noise=true , address=45.91.116.12:4226 , piece_id=CZDJLBP4LNXK6TZEF7QTTS2K2AI3YWL5GOOFFBZ4EDC6EDT5FKGA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.844780205 (381h11m48.71586324s) [noise=true , address=145.79.227.51:4409 , piece_id=2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.260685007 (381h11m49.131768042s) [noise=true , address=145.79.227.50:4375 , piece_id=U2ITQWUUV4B652CMHM7HSAZCZFOMLHQSRDXR2SWOQEDV4I36I2JQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.283553241 (381h11m49.154636276s) [noise=true , address=45.91.116.10:4083 , piece_id=QMO5GLTIPCL6URQKEYGFHBVMYYJGXGKXUFKZLYUN6DOA6I2AZ5LA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.283572278 (381h11m49.154655313s) [seg_pos={0 0}, upload=334532]: Segment reached optimal threshold of 13 pieces. Dec 21 12:40:49.283738254 (381h11m49.154821289s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.283744285 (381h11m49.15482732s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.283805843 (381h11m49.154888878s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.283820700 (381h11m49.154903735s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.284005161 (381h11m49.155088196s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.284011584 (381h11m49.155094619s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.284030430 (381h11m49.155113465s) [seg_pos={0 0}, upload=334532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:40:49.285004034 (381h11m49.156087069s) [upload=334532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:40:49.285034291 (381h11m49.156117337s) [upload=334532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:40:49.569439270 (381h11m49.440522305s) [upload=334532]: Done uploading segments... Dec 21 12:40:49.569454108 (381h11m49.440537153s) [upload=334532]: Upload finished. err: <nil> Duration: 29.240483998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....X] duration: 30.002157934s trace: https://jaeger.dev.storj.io/trace/3f772a8b8d28b331 FAIL (2025-12-21 12:41:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ) context deadline exceeded; piecestore: (Node ID: 124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, Piece ID: H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA) context deadline exceeded ======LOGS====== Dec 21 12:40:46.128080141 (381h11m45.999163186s) [upload=334532]: Starting upload Dec 21 12:40:46.128164041 (381h11m45.999247076s) [upload=334532]: Done starting upload Dec 21 12:40:46.128283829 (381h11m45.999366874s) [upload=334532]: Uploading segments... Dec 21 12:40:46.128447340 (381h11m45.999530385s) [upload=334532]: Got next segment. Inline: false Dec 21 12:40:46.128495151 (381h11m45.999578196s) [upload=334532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:40:46.128501985 (381h11m45.99958502s) [upload=334532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:40:46.371673158 (381h11m46.242756233s) [seg_pos={0 0}, upload=334532]: Begin upload segment... Dec 21 12:40:46.371738763 (381h11m46.242821818s) [seg_pos={0 0}, upload=334532]: Done begin upload segment. Dec 21 12:40:46.371779020 (381h11m46.242862085s) [noise=true , address=145.79.227.45:4230 , piece_id=Y7CIVZTY4J7WTF7DH25EGTQHMJHKEHCCHZMRDRCBQVWLL6PUSKEQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.371838442 (381h11m46.242921477s) [noise=true , address=45.91.116.14:4190 , piece_id=BTMYSIYFECGOBMLINMV4EGJBSDR7SBQTBYJPVDIF7IPRKV4ZBKJQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.372455948 (381h11m46.243538983s) [noise=true , address=145.79.227.47:4266 , piece_id=M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.372882260 (381h11m46.243965295s) [noise=true , address=145.79.227.51:4409 , piece_id=2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.373667365 (381h11m46.24475041s) [noise=true , address=45.91.116.11:4120 , piece_id=DZY76BJDMMIW32EG3Y5YXB62EBCININT6GOHJZICIYYXGFSQU6XA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.373687093 (381h11m46.244770138s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.374798799 (381h11m46.245881834s) [noise=true , address=45.91.116.10:4083 , piece_id=QMO5GLTIPCL6URQKEYGFHBVMYYJGXGKXUFKZLYUN6DOA6I2AZ5LA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.374958784 (381h11m46.246041829s) [noise=true , address=145.79.227.42:4122 , piece_id=H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.375881512 (381h11m46.246964567s) [noise=true , address=145.79.227.20:4437 , piece_id=FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.376081542 (381h11m46.247164587s) [noise=true , address=45.91.116.12:4226 , piece_id=CZDJLBP4LNXK6TZEF7QTTS2K2AI3YWL5GOOFFBZ4EDC6EDT5FKGA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.377087808 (381h11m46.248170843s) [noise=true , address=45.91.116.8:4045 , piece_id=H4LJDYLU7HQB2H5AVBOHUJFAE4C7ZJVMHWJWY3YRKDKXJJXRWSEA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.377286877 (381h11m46.248369912s) [noise=true , address=77.90.31.11:4555 , piece_id=NKSBSQ5PZAKZSTNT5UJ4XGZVOR4D7Y6VBNI3BJNNG5M4CWLSCQBQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.378029000 (381h11m46.249112045s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.378367575 (381h11m46.24945061s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.380863277 (381h11m46.251946322s) [noise=true , address=145.79.227.50:4375 , piece_id=U2ITQWUUV4B652CMHM7HSAZCZFOMLHQSRDXR2SWOQEDV4I36I2JQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:46.387072851 (381h11m46.258155897s) [noise=true , address=145.79.227.41:4085 , piece_id=QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ, seg_pos={0 0}, upload=334532]: Uploading piece... Dec 21 12:40:48.136762127 (381h11m48.007845162s) [upload=334532]: Next returned nil segment Dec 21 12:40:48.136777005 (381h11m48.00786004s) [upload=334532]: Waiting for error group managing segments... Dec 21 12:40:48.536674798 (381h11m48.407757843s) [noise=true , address=145.79.227.47:4266 , piece_id=M4YH27ZNLXBANTZP2XXWL6552BEZMT4O5FCGRCNY7XVMRJSJJWHA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.543204462 (381h11m48.414287507s) [noise=true , address=145.79.227.42:4122 , piece_id=H4CVC576NJNPZKQNKOGFFVTMOXWFCUPKNFPGM6VADIAZH53GRAWA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.580163226 (381h11m48.451246261s) [noise=true , address=45.91.116.8:4045 , piece_id=H4LJDYLU7HQB2H5AVBOHUJFAE4C7ZJVMHWJWY3YRKDKXJJXRWSEA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.581079802 (381h11m48.452162837s) [noise=true , address=145.79.227.20:4437 , piece_id=FRQLMVQNPBXNYPJ56VRDFVVNAEYJFE4CUMQO5HSDLMTHTGQCNGOQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.611779188 (381h11m48.482862223s) [noise=true , address=145.79.227.45:4230 , piece_id=Y7CIVZTY4J7WTF7DH25EGTQHMJHKEHCCHZMRDRCBQVWLL6PUSKEQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.639282628 (381h11m48.510365673s) [noise=true , address=145.79.227.41:4085 , piece_id=QHACKMWORPLUTRVFBXHNR4HXUZMKIMSG3XOQRKZLF3E6DNAJ6SNQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.666980178 (381h11m48.538063213s) [noise=true , address=45.91.116.14:4190 , piece_id=BTMYSIYFECGOBMLINMV4EGJBSDR7SBQTBYJPVDIF7IPRKV4ZBKJQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.667503785 (381h11m48.53858682s) [noise=true , address=45.91.116.11:4120 , piece_id=DZY76BJDMMIW32EG3Y5YXB62EBCININT6GOHJZICIYYXGFSQU6XA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.668450628 (381h11m48.539533673s) [noise=true , address=77.90.31.11:4555 , piece_id=NKSBSQ5PZAKZSTNT5UJ4XGZVOR4D7Y6VBNI3BJNNG5M4CWLSCQBQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.690686959 (381h11m48.561769994s) [noise=true , address=45.91.116.12:4226 , piece_id=CZDJLBP4LNXK6TZEF7QTTS2K2AI3YWL5GOOFFBZ4EDC6EDT5FKGA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:48.844780205 (381h11m48.71586324s) [noise=true , address=145.79.227.51:4409 , piece_id=2GSK2HVZWLLV77ARJZ6KJMCFQTXX6XDKM6K3AILNI4X2DXMRJDLQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.260685007 (381h11m49.131768042s) [noise=true , address=145.79.227.50:4375 , piece_id=U2ITQWUUV4B652CMHM7HSAZCZFOMLHQSRDXR2SWOQEDV4I36I2JQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.283553241 (381h11m49.154636276s) [noise=true , address=45.91.116.10:4083 , piece_id=QMO5GLTIPCL6URQKEYGFHBVMYYJGXGKXUFKZLYUN6DOA6I2AZ5LA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: <nil> Dec 21 12:40:49.283572278 (381h11m49.154655313s) [seg_pos={0 0}, upload=334532]: Segment reached optimal threshold of 13 pieces. Dec 21 12:40:49.283738254 (381h11m49.154821289s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.283744285 (381h11m49.15482732s) [noise=true , address=145.79.227.36:4519 , piece_id=WS5H4HHM4A7N23DB7DHQRLYCYCC2E2WUKLYVFI4Y2R6UWUPW35HA, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.283805843 (381h11m49.154888878s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.283820700 (381h11m49.154903735s) [noise=true , address=145.79.227.37:4566 , piece_id=I6JQMZBRYMCUROAWDMMKC7EPSWCID4OLB2TJLKDM36XUKYC66KVQ, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.284005161 (381h11m49.155088196s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 21 12:40:49.284011584 (381h11m49.155094619s) [noise=true , address=145.79.227.8:4330 , piece_id=JNREFDGVDQCEGVP5S3L7VFK4WWQFCCC6MFHVDYTLDVFXQ5WGYXCA, seg_pos={0 0}, upload=334532]: Long tail cancellation due to optimal threshold Dec 21 12:40:49.284030430 (381h11m49.155113465s) [seg_pos={0 0}, upload=334532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:40:49.285004034 (381h11m49.156087069s) [upload=334532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:40:49.285034291 (381h11m49.156117337s) [upload=334532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:40:49.569439270 (381h11m49.440522305s) [upload=334532]: Done uploading segments... Dec 21 12:40:49.569454108 (381h11m49.440537153s) [upload=334532]: Upload finished. err: <nil> Duration: 30.00217732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.....] duration: 20.863084207s trace: https://jaeger.dev.storj.io/trace/6bfb96fa1e4f5ae9 FAIL (2025-12-21 12:40:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:02.221651108 (381h7m2.092734163s) [upload=334468]: Starting upload Dec 21 12:36:02.221728085 (381h7m2.09281112s) [upload=334468]: Done starting upload Dec 21 12:36:02.221885735 (381h7m2.09296878s) [upload=334468]: Uploading segments... Dec 21 12:36:02.222203910 (381h7m2.093286965s) [upload=334468]: Got next segment. Inline: false Dec 21 12:36:02.222276358 (381h7m2.093359403s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:36:02.222282501 (381h7m2.093365546s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:36:02.472140144 (381h7m2.343223189s) [seg_pos={0 0}, upload=334468]: Begin upload segment... Dec 21 12:36:02.472224986 (381h7m2.343308031s) [seg_pos={0 0}, upload=334468]: Done begin upload segment. Dec 21 12:36:02.472271986 (381h7m2.343355041s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472289539 (381h7m2.343372575s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472342309 (381h7m2.343425355s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473032784 (381h7m2.344115849s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473664197 (381h7m2.344747232s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473671941 (381h7m2.344754986s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473733669 (381h7m2.344816714s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474540465 (381h7m2.3456235s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474984441 (381h7m2.346067486s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475429989 (381h7m2.346513024s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475937646 (381h7m2.347020691s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476420405 (381h7m2.34750346s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476479898 (381h7m2.347562943s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476980212 (381h7m2.348063257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477055014 (381h7m2.348138059s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477978492 (381h7m2.349061537s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:04.240587298 (381h7m4.111670333s) [upload=334468]: Next returned nil segment Dec 21 12:36:04.240603319 (381h7m4.111686364s) [upload=334468]: Waiting for error group managing segments... Dec 21 12:36:04.627886282 (381h7m4.498969327s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.653325331 (381h7m4.524408376s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.654272154 (381h7m4.525355199s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.687269946 (381h7m4.558352992s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.710916153 (381h7m4.581999198s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.718623408 (381h7m4.589706453s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.720772020 (381h7m4.591855065s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.723468173 (381h7m4.594551218s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.735291927 (381h7m4.606374972s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.756416442 (381h7m4.627499487s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.810969709 (381h7m4.682052754s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.961535047 (381h7m4.832618092s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009101921 (381h7m4.880184966s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009146616 (381h7m4.880229661s) [seg_pos={0 0}, upload=334468]: Segment reached optimal threshold of 13 pieces. Dec 21 12:36:05.009380441 (381h7m4.880463516s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 21 12:36:05.009390921 (381h7m4.880473966s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009551958 (381h7m4.880634993s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection Dec 21 12:36:05.009563059 (381h7m4.880646104s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009726631 (381h7m4.880809676s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: read tcp 188.166.225.170:36564->145.79.227.41:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:36:05.009736980 (381h7m4.880820025s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009768710 (381h7m4.880851756s) [seg_pos={0 0}, upload=334468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:36:05.010083229 (381h7m4.881166284s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:36:05.010091445 (381h7m4.88117449s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:36:05.303159068 (381h7m5.174242103s) [upload=334468]: Done uploading segments... Dec 21 12:36:05.303180039 (381h7m5.174263074s) [upload=334468]: Upload finished. err: <nil> Duration: 20.863132048s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X....] duration: 5.792965213s trace: https://jaeger.dev.storj.io/trace/6bfb96fa1e4f5ae9 FAIL (2025-12-21 12:39:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:02.221651108 (381h7m2.092734163s) [upload=334468]: Starting upload Dec 21 12:36:02.221728085 (381h7m2.09281112s) [upload=334468]: Done starting upload Dec 21 12:36:02.221885735 (381h7m2.09296878s) [upload=334468]: Uploading segments... Dec 21 12:36:02.222203910 (381h7m2.093286965s) [upload=334468]: Got next segment. Inline: false Dec 21 12:36:02.222276358 (381h7m2.093359403s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:36:02.222282501 (381h7m2.093365546s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:36:02.472140144 (381h7m2.343223189s) [seg_pos={0 0}, upload=334468]: Begin upload segment... Dec 21 12:36:02.472224986 (381h7m2.343308031s) [seg_pos={0 0}, upload=334468]: Done begin upload segment. Dec 21 12:36:02.472271986 (381h7m2.343355041s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472289539 (381h7m2.343372575s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472342309 (381h7m2.343425355s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473032784 (381h7m2.344115849s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473664197 (381h7m2.344747232s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473671941 (381h7m2.344754986s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473733669 (381h7m2.344816714s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474540465 (381h7m2.3456235s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474984441 (381h7m2.346067486s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475429989 (381h7m2.346513024s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475937646 (381h7m2.347020691s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476420405 (381h7m2.34750346s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476479898 (381h7m2.347562943s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476980212 (381h7m2.348063257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477055014 (381h7m2.348138059s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477978492 (381h7m2.349061537s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:04.240587298 (381h7m4.111670333s) [upload=334468]: Next returned nil segment Dec 21 12:36:04.240603319 (381h7m4.111686364s) [upload=334468]: Waiting for error group managing segments... Dec 21 12:36:04.627886282 (381h7m4.498969327s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.653325331 (381h7m4.524408376s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.654272154 (381h7m4.525355199s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.687269946 (381h7m4.558352992s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.710916153 (381h7m4.581999198s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.718623408 (381h7m4.589706453s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.720772020 (381h7m4.591855065s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.723468173 (381h7m4.594551218s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.735291927 (381h7m4.606374972s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.756416442 (381h7m4.627499487s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.810969709 (381h7m4.682052754s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.961535047 (381h7m4.832618092s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009101921 (381h7m4.880184966s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009146616 (381h7m4.880229661s) [seg_pos={0 0}, upload=334468]: Segment reached optimal threshold of 13 pieces. Dec 21 12:36:05.009380441 (381h7m4.880463516s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 21 12:36:05.009390921 (381h7m4.880473966s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009551958 (381h7m4.880634993s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection Dec 21 12:36:05.009563059 (381h7m4.880646104s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009726631 (381h7m4.880809676s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: read tcp 188.166.225.170:36564->145.79.227.41:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:36:05.009736980 (381h7m4.880820025s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009768710 (381h7m4.880851756s) [seg_pos={0 0}, upload=334468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:36:05.010083229 (381h7m4.881166284s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:36:05.010091445 (381h7m4.88117449s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:36:05.303159068 (381h7m5.174242103s) [upload=334468]: Done uploading segments... Dec 21 12:36:05.303180039 (381h7m5.174263074s) [upload=334468]: Upload finished. err: <nil> Duration: 5.792997956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X...] duration: 6.627574582s trace: https://jaeger.dev.storj.io/trace/6bfb96fa1e4f5ae9 FAIL (2025-12-21 12:39:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:02.221651108 (381h7m2.092734163s) [upload=334468]: Starting upload Dec 21 12:36:02.221728085 (381h7m2.09281112s) [upload=334468]: Done starting upload Dec 21 12:36:02.221885735 (381h7m2.09296878s) [upload=334468]: Uploading segments... Dec 21 12:36:02.222203910 (381h7m2.093286965s) [upload=334468]: Got next segment. Inline: false Dec 21 12:36:02.222276358 (381h7m2.093359403s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:36:02.222282501 (381h7m2.093365546s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:36:02.472140144 (381h7m2.343223189s) [seg_pos={0 0}, upload=334468]: Begin upload segment... Dec 21 12:36:02.472224986 (381h7m2.343308031s) [seg_pos={0 0}, upload=334468]: Done begin upload segment. Dec 21 12:36:02.472271986 (381h7m2.343355041s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472289539 (381h7m2.343372575s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472342309 (381h7m2.343425355s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473032784 (381h7m2.344115849s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473664197 (381h7m2.344747232s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473671941 (381h7m2.344754986s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473733669 (381h7m2.344816714s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474540465 (381h7m2.3456235s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474984441 (381h7m2.346067486s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475429989 (381h7m2.346513024s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475937646 (381h7m2.347020691s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476420405 (381h7m2.34750346s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476479898 (381h7m2.347562943s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476980212 (381h7m2.348063257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477055014 (381h7m2.348138059s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477978492 (381h7m2.349061537s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:04.240587298 (381h7m4.111670333s) [upload=334468]: Next returned nil segment Dec 21 12:36:04.240603319 (381h7m4.111686364s) [upload=334468]: Waiting for error group managing segments... Dec 21 12:36:04.627886282 (381h7m4.498969327s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.653325331 (381h7m4.524408376s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.654272154 (381h7m4.525355199s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.687269946 (381h7m4.558352992s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.710916153 (381h7m4.581999198s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.718623408 (381h7m4.589706453s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.720772020 (381h7m4.591855065s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.723468173 (381h7m4.594551218s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.735291927 (381h7m4.606374972s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.756416442 (381h7m4.627499487s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.810969709 (381h7m4.682052754s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.961535047 (381h7m4.832618092s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009101921 (381h7m4.880184966s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009146616 (381h7m4.880229661s) [seg_pos={0 0}, upload=334468]: Segment reached optimal threshold of 13 pieces. Dec 21 12:36:05.009380441 (381h7m4.880463516s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 21 12:36:05.009390921 (381h7m4.880473966s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009551958 (381h7m4.880634993s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection Dec 21 12:36:05.009563059 (381h7m4.880646104s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009726631 (381h7m4.880809676s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: read tcp 188.166.225.170:36564->145.79.227.41:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:36:05.009736980 (381h7m4.880820025s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009768710 (381h7m4.880851756s) [seg_pos={0 0}, upload=334468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:36:05.010083229 (381h7m4.881166284s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:36:05.010091445 (381h7m4.88117449s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:36:05.303159068 (381h7m5.174242103s) [upload=334468]: Done uploading segments... Dec 21 12:36:05.303180039 (381h7m5.174263074s) [upload=334468]: Upload finished. err: <nil> Duration: 6.627600851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [............XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X..] duration: 5.733396903s trace: https://jaeger.dev.storj.io/trace/6bfb96fa1e4f5ae9 FAIL (2025-12-21 12:38:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:02.221651108 (381h7m2.092734163s) [upload=334468]: Starting upload Dec 21 12:36:02.221728085 (381h7m2.09281112s) [upload=334468]: Done starting upload Dec 21 12:36:02.221885735 (381h7m2.09296878s) [upload=334468]: Uploading segments... Dec 21 12:36:02.222203910 (381h7m2.093286965s) [upload=334468]: Got next segment. Inline: false Dec 21 12:36:02.222276358 (381h7m2.093359403s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:36:02.222282501 (381h7m2.093365546s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:36:02.472140144 (381h7m2.343223189s) [seg_pos={0 0}, upload=334468]: Begin upload segment... Dec 21 12:36:02.472224986 (381h7m2.343308031s) [seg_pos={0 0}, upload=334468]: Done begin upload segment. Dec 21 12:36:02.472271986 (381h7m2.343355041s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472289539 (381h7m2.343372575s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472342309 (381h7m2.343425355s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473032784 (381h7m2.344115849s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473664197 (381h7m2.344747232s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473671941 (381h7m2.344754986s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473733669 (381h7m2.344816714s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474540465 (381h7m2.3456235s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474984441 (381h7m2.346067486s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475429989 (381h7m2.346513024s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475937646 (381h7m2.347020691s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476420405 (381h7m2.34750346s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476479898 (381h7m2.347562943s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476980212 (381h7m2.348063257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477055014 (381h7m2.348138059s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477978492 (381h7m2.349061537s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:04.240587298 (381h7m4.111670333s) [upload=334468]: Next returned nil segment Dec 21 12:36:04.240603319 (381h7m4.111686364s) [upload=334468]: Waiting for error group managing segments... Dec 21 12:36:04.627886282 (381h7m4.498969327s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.653325331 (381h7m4.524408376s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.654272154 (381h7m4.525355199s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.687269946 (381h7m4.558352992s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.710916153 (381h7m4.581999198s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.718623408 (381h7m4.589706453s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.720772020 (381h7m4.591855065s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.723468173 (381h7m4.594551218s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.735291927 (381h7m4.606374972s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.756416442 (381h7m4.627499487s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.810969709 (381h7m4.682052754s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.961535047 (381h7m4.832618092s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009101921 (381h7m4.880184966s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009146616 (381h7m4.880229661s) [seg_pos={0 0}, upload=334468]: Segment reached optimal threshold of 13 pieces. Dec 21 12:36:05.009380441 (381h7m4.880463516s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 21 12:36:05.009390921 (381h7m4.880473966s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009551958 (381h7m4.880634993s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection Dec 21 12:36:05.009563059 (381h7m4.880646104s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009726631 (381h7m4.880809676s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: read tcp 188.166.225.170:36564->145.79.227.41:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:36:05.009736980 (381h7m4.880820025s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009768710 (381h7m4.880851756s) [seg_pos={0 0}, upload=334468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:36:05.010083229 (381h7m4.881166284s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:36:05.010091445 (381h7m4.88117449s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:36:05.303159068 (381h7m5.174242103s) [upload=334468]: Done uploading segments... Dec 21 12:36:05.303180039 (381h7m5.174263074s) [upload=334468]: Upload finished. err: <nil> Duration: 5.733417403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.............XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X.] duration: 5.717119721s trace: https://jaeger.dev.storj.io/trace/6bfb96fa1e4f5ae9 FAIL (2025-12-21 12:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:02.221651108 (381h7m2.092734163s) [upload=334468]: Starting upload Dec 21 12:36:02.221728085 (381h7m2.09281112s) [upload=334468]: Done starting upload Dec 21 12:36:02.221885735 (381h7m2.09296878s) [upload=334468]: Uploading segments... Dec 21 12:36:02.222203910 (381h7m2.093286965s) [upload=334468]: Got next segment. Inline: false Dec 21 12:36:02.222276358 (381h7m2.093359403s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:36:02.222282501 (381h7m2.093365546s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:36:02.472140144 (381h7m2.343223189s) [seg_pos={0 0}, upload=334468]: Begin upload segment... Dec 21 12:36:02.472224986 (381h7m2.343308031s) [seg_pos={0 0}, upload=334468]: Done begin upload segment. Dec 21 12:36:02.472271986 (381h7m2.343355041s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472289539 (381h7m2.343372575s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472342309 (381h7m2.343425355s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473032784 (381h7m2.344115849s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473664197 (381h7m2.344747232s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473671941 (381h7m2.344754986s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473733669 (381h7m2.344816714s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474540465 (381h7m2.3456235s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474984441 (381h7m2.346067486s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475429989 (381h7m2.346513024s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475937646 (381h7m2.347020691s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476420405 (381h7m2.34750346s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476479898 (381h7m2.347562943s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476980212 (381h7m2.348063257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477055014 (381h7m2.348138059s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477978492 (381h7m2.349061537s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:04.240587298 (381h7m4.111670333s) [upload=334468]: Next returned nil segment Dec 21 12:36:04.240603319 (381h7m4.111686364s) [upload=334468]: Waiting for error group managing segments... Dec 21 12:36:04.627886282 (381h7m4.498969327s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.653325331 (381h7m4.524408376s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.654272154 (381h7m4.525355199s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.687269946 (381h7m4.558352992s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.710916153 (381h7m4.581999198s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.718623408 (381h7m4.589706453s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.720772020 (381h7m4.591855065s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.723468173 (381h7m4.594551218s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.735291927 (381h7m4.606374972s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.756416442 (381h7m4.627499487s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.810969709 (381h7m4.682052754s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.961535047 (381h7m4.832618092s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009101921 (381h7m4.880184966s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009146616 (381h7m4.880229661s) [seg_pos={0 0}, upload=334468]: Segment reached optimal threshold of 13 pieces. Dec 21 12:36:05.009380441 (381h7m4.880463516s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 21 12:36:05.009390921 (381h7m4.880473966s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009551958 (381h7m4.880634993s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection Dec 21 12:36:05.009563059 (381h7m4.880646104s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009726631 (381h7m4.880809676s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: read tcp 188.166.225.170:36564->145.79.227.41:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:36:05.009736980 (381h7m4.880820025s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009768710 (381h7m4.880851756s) [seg_pos={0 0}, upload=334468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:36:05.010083229 (381h7m4.881166284s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:36:05.010091445 (381h7m4.88117449s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:36:05.303159068 (381h7m5.174242103s) [upload=334468]: Done uploading segments... Dec 21 12:36:05.303180039 (381h7m5.174263074s) [upload=334468]: Upload finished. err: <nil> Duration: 5.71714547s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..............XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..X] duration: 30.003365482s trace: https://jaeger.dev.storj.io/trace/6bfb96fa1e4f5ae9 FAIL (2025-12-21 12:36:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6:02.221651108 (381h7m2.092734163s) [upload=334468]: Starting upload Dec 21 12:36:02.221728085 (381h7m2.09281112s) [upload=334468]: Done starting upload Dec 21 12:36:02.221885735 (381h7m2.09296878s) [upload=334468]: Uploading segments... Dec 21 12:36:02.222203910 (381h7m2.093286965s) [upload=334468]: Got next segment. Inline: false Dec 21 12:36:02.222276358 (381h7m2.093359403s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:36:02.222282501 (381h7m2.093365546s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:36:02.472140144 (381h7m2.343223189s) [seg_pos={0 0}, upload=334468]: Begin upload segment... Dec 21 12:36:02.472224986 (381h7m2.343308031s) [seg_pos={0 0}, upload=334468]: Done begin upload segment. Dec 21 12:36:02.472271986 (381h7m2.343355041s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472289539 (381h7m2.343372575s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.472342309 (381h7m2.343425355s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473032784 (381h7m2.344115849s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473664197 (381h7m2.344747232s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473671941 (381h7m2.344754986s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.473733669 (381h7m2.344816714s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474540465 (381h7m2.3456235s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.474984441 (381h7m2.346067486s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475429989 (381h7m2.346513024s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.475937646 (381h7m2.347020691s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476420405 (381h7m2.34750346s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476479898 (381h7m2.347562943s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.476980212 (381h7m2.348063257s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477055014 (381h7m2.348138059s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:02.477978492 (381h7m2.349061537s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Uploading piece... Dec 21 12:36:04.240587298 (381h7m4.111670333s) [upload=334468]: Next returned nil segment Dec 21 12:36:04.240603319 (381h7m4.111686364s) [upload=334468]: Waiting for error group managing segments... Dec 21 12:36:04.627886282 (381h7m4.498969327s) [noise=true , address=145.79.227.11:4731 , piece_id=AZFFHHAZIQ6HPJE5263A37GBEWTV7OAUQ5XXPVEG57L3RJMIX7FQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.653325331 (381h7m4.524408376s) [noise=true , address=145.79.227.40:4051 , piece_id=KLM4OBQ7QOAQ5UGQEPPQF3J7Z2J5V2ICCJZKUYUTM3ZZEI26AG5A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.654272154 (381h7m4.525355199s) [noise=true , address=145.79.227.29:4941 , piece_id=QWBKF3YDFC6Y43R5KJTPFERE3T3YZRKLLZ37MJ26WS2NSOZWWUDQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.687269946 (381h7m4.558352992s) [noise=true , address=145.79.227.4:4186 , piece_id=5XJGO6JBZFVOYODTGOUAL56OG2WXZIKNRUIQSITJAPTNQZBXIB4A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.710916153 (381h7m4.581999198s) [noise=true , address=145.79.227.27:4691 , piece_id=DGC7CE6HDYLW7ZJ6PVA5TBZPRZIGOILJN5I3DGLQBLN4NA76PJAA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.718623408 (381h7m4.589706453s) [noise=true , address=45.91.116.11:4118 , piece_id=2LEZDX5DY4PSKP2QNJEECVXHBPBZFVLYGCOLWGL7CHRAJF2QSEZA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.720772020 (381h7m4.591855065s) [noise=true , address=45.91.116.8:4048 , piece_id=64P2PXLHBOP427NFU7CVSRF4Y5Z6D5LVGUNEKO55JKEPI2X7NHSA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.723468173 (381h7m4.594551218s) [noise=true , address=145.79.227.20:4439 , piece_id=TSK7K5TN5Z3SUWZCZ5JZD2767ZWTWROKA43CDAVL3PYU2Q3MWSTQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.735291927 (381h7m4.606374972s) [noise=true , address=45.91.116.10:4082 , piece_id=M22BGF3HFK2I3SOTIZYJFPRHJIJ3N4XN4PGRJSFCKFN27ENX2LNQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.756416442 (381h7m4.627499487s) [noise=true , address=145.79.227.49:4337 , piece_id=HJFSY77L7ZA6CWYIS4SIJISQAAVDK7OAOW42M562VJJHCKNBPWWA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.810969709 (381h7m4.682052754s) [noise=true , address=45.91.116.14:4192 , piece_id=HKK5MMMFJ7XMEPEF6ZIIGFXRNEYVDGIO4I2RYKFZE2JDQT7U6KQA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:04.961535047 (381h7m4.832618092s) [noise=true , address=145.79.227.51:4409 , piece_id=LT3DDA7RDOVSWCI27H22VKO272ZRMWA6HQGG26YL5QET4FHQL25A, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009101921 (381h7m4.880184966s) [noise=true , address=77.90.31.14:4662 , piece_id=ZJEFIJNHLYG6NGAG6HYUX2TCLG5TISFHGG76GHXPWKP4AZ52HTBA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: <nil> Dec 21 12:36:05.009146616 (381h7m4.880229661s) [seg_pos={0 0}, upload=334468]: Segment reached optimal threshold of 13 pieces. Dec 21 12:36:05.009380441 (381h7m4.880463516s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 21 12:36:05.009390921 (381h7m4.880473966s) [noise=true , address=45.91.116.5:4299 , piece_id=BPFWXMOF6TPGR7XXT6PWEFRDP3IIVTAXKYE5YW2WTVY5LS25E6NQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009551958 (381h7m4.880634993s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk): protocol: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33836->145.79.227.32:4834: use of closed network connection Dec 21 12:36:05.009563059 (381h7m4.880646104s) [noise=true , address=145.79.227.32:4834 , piece_id=KOL75HEGV2MW2PSPNNHACXVPWSFGVMLRBIYUOBUQZPKYT56AW5JA, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009726631 (381h7m4.880809676s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: read tcp 188.166.225.170:36564->145.79.227.41:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:36:05.009736980 (381h7m4.880820025s) [noise=true , address=145.79.227.41:4085 , piece_id=3JVT3EX7IBCU5XP3PJRWQKB3UC5JE64D7SH7C3JY4Y7GQC3GQAAQ, seg_pos={0 0}, upload=334468]: Long tail cancellation due to optimal threshold Dec 21 12:36:05.009768710 (381h7m4.880851756s) [seg_pos={0 0}, upload=334468]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:36:05.010083229 (381h7m4.881166284s) [upload=334468]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:36:05.010091445 (381h7m4.88117449s) [upload=334468]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:36:05.303159068 (381h7m5.174242103s) [upload=334468]: Done uploading segments... Dec 21 12:36:05.303180039 (381h7m5.174263074s) [upload=334468]: Upload finished. err: <nil> Duration: 30.003389537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...............XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X..] duration: 6.306029853s trace: https://jaeger.dev.storj.io/trace/395dacfad8cd67bf FAIL (2025-12-21 12:35:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:33:10.842467443 (381h4m10.713550488s) [upload=334427]: Starting upload Dec 21 12:33:10.842546243 (381h4m10.713629278s) [upload=334427]: Done starting upload Dec 21 12:33:10.842644801 (381h4m10.713727846s) [upload=334427]: Uploading segments... Dec 21 12:33:10.842848549 (381h4m10.713931594s) [upload=334427]: Got next segment. Inline: false Dec 21 12:33:10.842893905 (381h4m10.71397694s) [upload=334427]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:33:10.842899205 (381h4m10.71398225s) [upload=334427]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:33:11.087120938 (381h4m10.958203973s) [seg_pos={0 0}, upload=334427]: Begin upload segment... Dec 21 12:33:11.087172867 (381h4m10.958255902s) [seg_pos={0 0}, upload=334427]: Done begin upload segment. Dec 21 12:33:11.087209036 (381h4m10.958292071s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.087241227 (381h4m10.958324262s) [noise=true , address=145.79.227.32:4835 , piece_id=H26TOKA75I4YIKQDL7DCRU3KJFIRSBJLWMTNLVRFRVQDZXG4OJTA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088171659 (381h4m10.959254704s) [noise=true , address=145.79.227.42:4123 , piece_id=ORTW5KNMFSQKRXXSJYLX3TKPX7AALEJIJCOIF65ZYUX7TPYTJASA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088631284 (381h4m10.959714319s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZPRVNKCKXLWQ33MA7FG76CJCGJKLVP6PRJLV5TFSTIMRITIX5HQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088975410 (381h4m10.960058455s) [noise=true , address=145.79.227.47:4266 , piece_id=MN6NQTVICRHJSDUKF7Q6RM5N4CCMQ5SJPYWQ5AFV73Y4YLUKTOYA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.089083185 (381h4m10.96016623s) [noise=true , address=145.79.227.48:4302 , piece_id=LBH3OHL6KBQF53NLCHRYE6MITBF2G677ADJDUKLVYWP5TIVAFR4A, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.090448845 (381h4m10.96153188s) [noise=true , address=145.79.227.39:4014 , piece_id=XLLVDNBRT5HNDQDKOJ5WPC7YXPSEVHW7W5VMUX4KEXGRIV5WVQHA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.091439892 (381h4m10.962522927s) [noise=true , address=145.79.227.49:4337 , piece_id=ENA2BTOYC3GPDXP2Z54TPCAWUDYFFD6WRJAKQCZPR2GMDQHJAQPQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.091503313 (381h4m10.962586358s) [noise=true , address=145.79.227.10:4402 , piece_id=5EROC47KJVASFF5HMCP5LADA4GBHOJ673ULHU4MN3ZYFJDR5P2SA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.092067407 (381h4m10.963150462s) [noise=true , address=45.91.116.14:4192 , piece_id=5NWGWEPI2KVAJ6LWFUWG7NT6U4EJZOLZO5CCOTEQCMJVNMUEHSGQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.092780485 (381h4m10.96386353s) [noise=true , address=145.79.227.27:4691 , piece_id=6LF3XPAMT4YEM5JAYAPLFTQNO3XTC7JV7JYUMQDOHXCR6LE26YTA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.093372452 (381h4m10.964455497s) [noise=true , address=145.79.227.15:4875 , piece_id=OWM4DUF6FXBJPFWJ2LVXZMDWJLAOHC2IZJFDNZ6NIURHTF33R27Q, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.093576289 (381h4m10.964659334s) [noise=true , address=145.79.227.38:4978 , piece_id=2QEMGPT36KLJ6NN6KPB2EJDIYZ6PZJYVYCCBEOE3A7QCAKATAPXQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.094346667 (381h4m10.965429712s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.094873530 (381h4m10.965956565s) [noise=true , address=77.90.31.18:4805 , piece_id=P5E6TC4AZ44XNTSCB5HA5WCM3IYSWK3LHFRKCZAUHWL5MEQNPD2Q, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.095254175 (381h4m10.96633722s) [noise=true , address=145.79.227.20:4439 , piece_id=KYM6YJVCQZY44TFOM7W4J2LVS4HAILC6W22OTSKVETPBVVCRKV2A, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:12.210657673 (381h4m12.081740708s) [noise=true , address=77.90.31.18:4805 , piece_id=P5E6TC4AZ44XNTSCB5HA5WCM3IYSWK3LHFRKCZAUHWL5MEQNPD2Q, seg_pos={0 0}, upload=334427]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 21 12:33:12.934253473 (381h4m12.805336508s) [upload=334427]: Next returned nil segment Dec 21 12:33:12.934267961 (381h4m12.805350996s) [upload=334427]: Waiting for error group managing segments... Dec 21 12:33:13.239885961 (381h4m13.110969006s) [noise=true , address=145.79.227.10:4402 , piece_id=5EROC47KJVASFF5HMCP5LADA4GBHOJ673ULHU4MN3ZYFJDR5P2SA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.266474700 (381h4m13.137557735s) [noise=true , address=145.79.227.48:4302 , piece_id=LBH3OHL6KBQF53NLCHRYE6MITBF2G677ADJDUKLVYWP5TIVAFR4A, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.310795533 (381h4m13.181878568s) [noise=true , address=145.79.227.39:4014 , piece_id=XLLVDNBRT5HNDQDKOJ5WPC7YXPSEVHW7W5VMUX4KEXGRIV5WVQHA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.327143241 (381h4m13.198226286s) [noise=true , address=145.79.227.32:4835 , piece_id=H26TOKA75I4YIKQDL7DCRU3KJFIRSBJLWMTNLVRFRVQDZXG4OJTA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.379655332 (381h4m13.250738367s) [noise=true , address=145.79.227.42:4123 , piece_id=ORTW5KNMFSQKRXXSJYLX3TKPX7AALEJIJCOIF65ZYUX7TPYTJASA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.386189685 (381h4m13.25727273s) [noise=true , address=145.79.227.27:4691 , piece_id=6LF3XPAMT4YEM5JAYAPLFTQNO3XTC7JV7JYUMQDOHXCR6LE26YTA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.387002632 (381h4m13.258085677s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZPRVNKCKXLWQ33MA7FG76CJCGJKLVP6PRJLV5TFSTIMRITIX5HQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.396261733 (381h4m13.267344768s) [noise=true , address=145.79.227.15:4875 , piece_id=OWM4DUF6FXBJPFWJ2LVXZMDWJLAOHC2IZJFDNZ6NIURHTF33R27Q, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.417834511 (381h4m13.288917566s) [noise=true , address=45.91.116.14:4192 , piece_id=5NWGWEPI2KVAJ6LWFUWG7NT6U4EJZOLZO5CCOTEQCMJVNMUEHSGQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.585552159 (381h4m13.456635204s) [noise=true , address=145.79.227.20:4439 , piece_id=KYM6YJVCQZY44TFOM7W4J2LVS4HAILC6W22OTSKVETPBVVCRKV2A, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.726599364 (381h4m13.597682399s) [noise=true , address=145.79.227.38:4978 , piece_id=2QEMGPT36KLJ6NN6KPB2EJDIYZ6PZJYVYCCBEOE3A7QCAKATAPXQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.276779516 (381h4m14.147862551s) [noise=true , address=145.79.227.47:4266 , piece_id=MN6NQTVICRHJSDUKF7Q6RM5N4CCMQ5SJPYWQ5AFV73Y4YLUKTOYA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.427259020 (381h4m14.298342055s) [noise=true , address=145.79.227.49:4337 , piece_id=ENA2BTOYC3GPDXP2Z54TPCAWUDYFFD6WRJAKQCZPR2GMDQHJAQPQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.427283867 (381h4m14.298366912s) [seg_pos={0 0}, upload=334427]: Segment reached optimal threshold of 13 pieces. Dec 21 12:33:14.427589459 (381h4m14.298672494s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 21 12:33:14.427598365 (381h4m14.298681401s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Long tail cancellation due to optimal threshold Dec 21 12:33:14.427977878 (381h4m14.299060923s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 21 12:33:14.427987356 (381h4m14.299070391s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Long tail cancellation due to optimal threshold Dec 21 12:33:14.428082487 (381h4m14.299165522s) [seg_pos={0 0}, upload=334427]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:33:14.428189882 (381h4m14.299272917s) [upload=334427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:33:14.428196234 (381h4m14.299279279s) [upload=334427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:33:14.725380605 (381h4m14.59646365s) [upload=334427]: Done uploading segments... Dec 21 12:33:14.725423627 (381h4m14.596506672s) [upload=334427]: Upload finished. err: <nil> Duration: 6.306061304s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [................XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X.] duration: 21.182640024s trace: https://jaeger.dev.storj.io/trace/395dacfad8cd67bf FAIL (2025-12-21 12:34:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:33:10.842467443 (381h4m10.713550488s) [upload=334427]: Starting upload Dec 21 12:33:10.842546243 (381h4m10.713629278s) [upload=334427]: Done starting upload Dec 21 12:33:10.842644801 (381h4m10.713727846s) [upload=334427]: Uploading segments... Dec 21 12:33:10.842848549 (381h4m10.713931594s) [upload=334427]: Got next segment. Inline: false Dec 21 12:33:10.842893905 (381h4m10.71397694s) [upload=334427]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:33:10.842899205 (381h4m10.71398225s) [upload=334427]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:33:11.087120938 (381h4m10.958203973s) [seg_pos={0 0}, upload=334427]: Begin upload segment... Dec 21 12:33:11.087172867 (381h4m10.958255902s) [seg_pos={0 0}, upload=334427]: Done begin upload segment. Dec 21 12:33:11.087209036 (381h4m10.958292071s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.087241227 (381h4m10.958324262s) [noise=true , address=145.79.227.32:4835 , piece_id=H26TOKA75I4YIKQDL7DCRU3KJFIRSBJLWMTNLVRFRVQDZXG4OJTA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088171659 (381h4m10.959254704s) [noise=true , address=145.79.227.42:4123 , piece_id=ORTW5KNMFSQKRXXSJYLX3TKPX7AALEJIJCOIF65ZYUX7TPYTJASA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088631284 (381h4m10.959714319s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZPRVNKCKXLWQ33MA7FG76CJCGJKLVP6PRJLV5TFSTIMRITIX5HQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088975410 (381h4m10.960058455s) [noise=true , address=145.79.227.47:4266 , piece_id=MN6NQTVICRHJSDUKF7Q6RM5N4CCMQ5SJPYWQ5AFV73Y4YLUKTOYA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.089083185 (381h4m10.96016623s) [noise=true , address=145.79.227.48:4302 , piece_id=LBH3OHL6KBQF53NLCHRYE6MITBF2G677ADJDUKLVYWP5TIVAFR4A, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.090448845 (381h4m10.96153188s) [noise=true , address=145.79.227.39:4014 , piece_id=XLLVDNBRT5HNDQDKOJ5WPC7YXPSEVHW7W5VMUX4KEXGRIV5WVQHA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.091439892 (381h4m10.962522927s) [noise=true , address=145.79.227.49:4337 , piece_id=ENA2BTOYC3GPDXP2Z54TPCAWUDYFFD6WRJAKQCZPR2GMDQHJAQPQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.091503313 (381h4m10.962586358s) [noise=true , address=145.79.227.10:4402 , piece_id=5EROC47KJVASFF5HMCP5LADA4GBHOJ673ULHU4MN3ZYFJDR5P2SA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.092067407 (381h4m10.963150462s) [noise=true , address=45.91.116.14:4192 , piece_id=5NWGWEPI2KVAJ6LWFUWG7NT6U4EJZOLZO5CCOTEQCMJVNMUEHSGQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.092780485 (381h4m10.96386353s) [noise=true , address=145.79.227.27:4691 , piece_id=6LF3XPAMT4YEM5JAYAPLFTQNO3XTC7JV7JYUMQDOHXCR6LE26YTA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.093372452 (381h4m10.964455497s) [noise=true , address=145.79.227.15:4875 , piece_id=OWM4DUF6FXBJPFWJ2LVXZMDWJLAOHC2IZJFDNZ6NIURHTF33R27Q, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.093576289 (381h4m10.964659334s) [noise=true , address=145.79.227.38:4978 , piece_id=2QEMGPT36KLJ6NN6KPB2EJDIYZ6PZJYVYCCBEOE3A7QCAKATAPXQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.094346667 (381h4m10.965429712s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.094873530 (381h4m10.965956565s) [noise=true , address=77.90.31.18:4805 , piece_id=P5E6TC4AZ44XNTSCB5HA5WCM3IYSWK3LHFRKCZAUHWL5MEQNPD2Q, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.095254175 (381h4m10.96633722s) [noise=true , address=145.79.227.20:4439 , piece_id=KYM6YJVCQZY44TFOM7W4J2LVS4HAILC6W22OTSKVETPBVVCRKV2A, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:12.210657673 (381h4m12.081740708s) [noise=true , address=77.90.31.18:4805 , piece_id=P5E6TC4AZ44XNTSCB5HA5WCM3IYSWK3LHFRKCZAUHWL5MEQNPD2Q, seg_pos={0 0}, upload=334427]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 21 12:33:12.934253473 (381h4m12.805336508s) [upload=334427]: Next returned nil segment Dec 21 12:33:12.934267961 (381h4m12.805350996s) [upload=334427]: Waiting for error group managing segments... Dec 21 12:33:13.239885961 (381h4m13.110969006s) [noise=true , address=145.79.227.10:4402 , piece_id=5EROC47KJVASFF5HMCP5LADA4GBHOJ673ULHU4MN3ZYFJDR5P2SA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.266474700 (381h4m13.137557735s) [noise=true , address=145.79.227.48:4302 , piece_id=LBH3OHL6KBQF53NLCHRYE6MITBF2G677ADJDUKLVYWP5TIVAFR4A, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.310795533 (381h4m13.181878568s) [noise=true , address=145.79.227.39:4014 , piece_id=XLLVDNBRT5HNDQDKOJ5WPC7YXPSEVHW7W5VMUX4KEXGRIV5WVQHA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.327143241 (381h4m13.198226286s) [noise=true , address=145.79.227.32:4835 , piece_id=H26TOKA75I4YIKQDL7DCRU3KJFIRSBJLWMTNLVRFRVQDZXG4OJTA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.379655332 (381h4m13.250738367s) [noise=true , address=145.79.227.42:4123 , piece_id=ORTW5KNMFSQKRXXSJYLX3TKPX7AALEJIJCOIF65ZYUX7TPYTJASA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.386189685 (381h4m13.25727273s) [noise=true , address=145.79.227.27:4691 , piece_id=6LF3XPAMT4YEM5JAYAPLFTQNO3XTC7JV7JYUMQDOHXCR6LE26YTA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.387002632 (381h4m13.258085677s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZPRVNKCKXLWQ33MA7FG76CJCGJKLVP6PRJLV5TFSTIMRITIX5HQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.396261733 (381h4m13.267344768s) [noise=true , address=145.79.227.15:4875 , piece_id=OWM4DUF6FXBJPFWJ2LVXZMDWJLAOHC2IZJFDNZ6NIURHTF33R27Q, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.417834511 (381h4m13.288917566s) [noise=true , address=45.91.116.14:4192 , piece_id=5NWGWEPI2KVAJ6LWFUWG7NT6U4EJZOLZO5CCOTEQCMJVNMUEHSGQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.585552159 (381h4m13.456635204s) [noise=true , address=145.79.227.20:4439 , piece_id=KYM6YJVCQZY44TFOM7W4J2LVS4HAILC6W22OTSKVETPBVVCRKV2A, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.726599364 (381h4m13.597682399s) [noise=true , address=145.79.227.38:4978 , piece_id=2QEMGPT36KLJ6NN6KPB2EJDIYZ6PZJYVYCCBEOE3A7QCAKATAPXQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.276779516 (381h4m14.147862551s) [noise=true , address=145.79.227.47:4266 , piece_id=MN6NQTVICRHJSDUKF7Q6RM5N4CCMQ5SJPYWQ5AFV73Y4YLUKTOYA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.427259020 (381h4m14.298342055s) [noise=true , address=145.79.227.49:4337 , piece_id=ENA2BTOYC3GPDXP2Z54TPCAWUDYFFD6WRJAKQCZPR2GMDQHJAQPQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.427283867 (381h4m14.298366912s) [seg_pos={0 0}, upload=334427]: Segment reached optimal threshold of 13 pieces. Dec 21 12:33:14.427589459 (381h4m14.298672494s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 21 12:33:14.427598365 (381h4m14.298681401s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Long tail cancellation due to optimal threshold Dec 21 12:33:14.427977878 (381h4m14.299060923s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 21 12:33:14.427987356 (381h4m14.299070391s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Long tail cancellation due to optimal threshold Dec 21 12:33:14.428082487 (381h4m14.299165522s) [seg_pos={0 0}, upload=334427]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:33:14.428189882 (381h4m14.299272917s) [upload=334427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:33:14.428196234 (381h4m14.299279279s) [upload=334427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:33:14.725380605 (381h4m14.59646365s) [upload=334427]: Done uploading segments... Dec 21 12:33:14.725423627 (381h4m14.596506672s) [upload=334427]: Upload finished. err: <nil> Duration: 21.182674442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.................XX.X..X.X..X.XXX.X.X..X.........X.X.X.....X] duration: 30.002679886s trace: https://jaeger.dev.storj.io/trace/395dacfad8cd67bf FAIL (2025-12-21 12:33:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:33:10.842467443 (381h4m10.713550488s) [upload=334427]: Starting upload Dec 21 12:33:10.842546243 (381h4m10.713629278s) [upload=334427]: Done starting upload Dec 21 12:33:10.842644801 (381h4m10.713727846s) [upload=334427]: Uploading segments... Dec 21 12:33:10.842848549 (381h4m10.713931594s) [upload=334427]: Got next segment. Inline: false Dec 21 12:33:10.842893905 (381h4m10.71397694s) [upload=334427]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:33:10.842899205 (381h4m10.71398225s) [upload=334427]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:33:11.087120938 (381h4m10.958203973s) [seg_pos={0 0}, upload=334427]: Begin upload segment... Dec 21 12:33:11.087172867 (381h4m10.958255902s) [seg_pos={0 0}, upload=334427]: Done begin upload segment. Dec 21 12:33:11.087209036 (381h4m10.958292071s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.087241227 (381h4m10.958324262s) [noise=true , address=145.79.227.32:4835 , piece_id=H26TOKA75I4YIKQDL7DCRU3KJFIRSBJLWMTNLVRFRVQDZXG4OJTA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088171659 (381h4m10.959254704s) [noise=true , address=145.79.227.42:4123 , piece_id=ORTW5KNMFSQKRXXSJYLX3TKPX7AALEJIJCOIF65ZYUX7TPYTJASA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088631284 (381h4m10.959714319s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZPRVNKCKXLWQ33MA7FG76CJCGJKLVP6PRJLV5TFSTIMRITIX5HQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.088975410 (381h4m10.960058455s) [noise=true , address=145.79.227.47:4266 , piece_id=MN6NQTVICRHJSDUKF7Q6RM5N4CCMQ5SJPYWQ5AFV73Y4YLUKTOYA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.089083185 (381h4m10.96016623s) [noise=true , address=145.79.227.48:4302 , piece_id=LBH3OHL6KBQF53NLCHRYE6MITBF2G677ADJDUKLVYWP5TIVAFR4A, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.090448845 (381h4m10.96153188s) [noise=true , address=145.79.227.39:4014 , piece_id=XLLVDNBRT5HNDQDKOJ5WPC7YXPSEVHW7W5VMUX4KEXGRIV5WVQHA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.091439892 (381h4m10.962522927s) [noise=true , address=145.79.227.49:4337 , piece_id=ENA2BTOYC3GPDXP2Z54TPCAWUDYFFD6WRJAKQCZPR2GMDQHJAQPQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.091503313 (381h4m10.962586358s) [noise=true , address=145.79.227.10:4402 , piece_id=5EROC47KJVASFF5HMCP5LADA4GBHOJ673ULHU4MN3ZYFJDR5P2SA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.092067407 (381h4m10.963150462s) [noise=true , address=45.91.116.14:4192 , piece_id=5NWGWEPI2KVAJ6LWFUWG7NT6U4EJZOLZO5CCOTEQCMJVNMUEHSGQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.092780485 (381h4m10.96386353s) [noise=true , address=145.79.227.27:4691 , piece_id=6LF3XPAMT4YEM5JAYAPLFTQNO3XTC7JV7JYUMQDOHXCR6LE26YTA, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.093372452 (381h4m10.964455497s) [noise=true , address=145.79.227.15:4875 , piece_id=OWM4DUF6FXBJPFWJ2LVXZMDWJLAOHC2IZJFDNZ6NIURHTF33R27Q, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.093576289 (381h4m10.964659334s) [noise=true , address=145.79.227.38:4978 , piece_id=2QEMGPT36KLJ6NN6KPB2EJDIYZ6PZJYVYCCBEOE3A7QCAKATAPXQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.094346667 (381h4m10.965429712s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.094873530 (381h4m10.965956565s) [noise=true , address=77.90.31.18:4805 , piece_id=P5E6TC4AZ44XNTSCB5HA5WCM3IYSWK3LHFRKCZAUHWL5MEQNPD2Q, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:11.095254175 (381h4m10.96633722s) [noise=true , address=145.79.227.20:4439 , piece_id=KYM6YJVCQZY44TFOM7W4J2LVS4HAILC6W22OTSKVETPBVVCRKV2A, seg_pos={0 0}, upload=334427]: Uploading piece... Dec 21 12:33:12.210657673 (381h4m12.081740708s) [noise=true , address=77.90.31.18:4805 , piece_id=P5E6TC4AZ44XNTSCB5HA5WCM3IYSWK3LHFRKCZAUHWL5MEQNPD2Q, seg_pos={0 0}, upload=334427]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 21 12:33:12.934253473 (381h4m12.805336508s) [upload=334427]: Next returned nil segment Dec 21 12:33:12.934267961 (381h4m12.805350996s) [upload=334427]: Waiting for error group managing segments... Dec 21 12:33:13.239885961 (381h4m13.110969006s) [noise=true , address=145.79.227.10:4402 , piece_id=5EROC47KJVASFF5HMCP5LADA4GBHOJ673ULHU4MN3ZYFJDR5P2SA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.266474700 (381h4m13.137557735s) [noise=true , address=145.79.227.48:4302 , piece_id=LBH3OHL6KBQF53NLCHRYE6MITBF2G677ADJDUKLVYWP5TIVAFR4A, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.310795533 (381h4m13.181878568s) [noise=true , address=145.79.227.39:4014 , piece_id=XLLVDNBRT5HNDQDKOJ5WPC7YXPSEVHW7W5VMUX4KEXGRIV5WVQHA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.327143241 (381h4m13.198226286s) [noise=true , address=145.79.227.32:4835 , piece_id=H26TOKA75I4YIKQDL7DCRU3KJFIRSBJLWMTNLVRFRVQDZXG4OJTA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.379655332 (381h4m13.250738367s) [noise=true , address=145.79.227.42:4123 , piece_id=ORTW5KNMFSQKRXXSJYLX3TKPX7AALEJIJCOIF65ZYUX7TPYTJASA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.386189685 (381h4m13.25727273s) [noise=true , address=145.79.227.27:4691 , piece_id=6LF3XPAMT4YEM5JAYAPLFTQNO3XTC7JV7JYUMQDOHXCR6LE26YTA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.387002632 (381h4m13.258085677s) [noise=true , address=145.79.227.50:4373 , piece_id=ZZPRVNKCKXLWQ33MA7FG76CJCGJKLVP6PRJLV5TFSTIMRITIX5HQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.396261733 (381h4m13.267344768s) [noise=true , address=145.79.227.15:4875 , piece_id=OWM4DUF6FXBJPFWJ2LVXZMDWJLAOHC2IZJFDNZ6NIURHTF33R27Q, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.417834511 (381h4m13.288917566s) [noise=true , address=45.91.116.14:4192 , piece_id=5NWGWEPI2KVAJ6LWFUWG7NT6U4EJZOLZO5CCOTEQCMJVNMUEHSGQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.585552159 (381h4m13.456635204s) [noise=true , address=145.79.227.20:4439 , piece_id=KYM6YJVCQZY44TFOM7W4J2LVS4HAILC6W22OTSKVETPBVVCRKV2A, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:13.726599364 (381h4m13.597682399s) [noise=true , address=145.79.227.38:4978 , piece_id=2QEMGPT36KLJ6NN6KPB2EJDIYZ6PZJYVYCCBEOE3A7QCAKATAPXQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.276779516 (381h4m14.147862551s) [noise=true , address=145.79.227.47:4266 , piece_id=MN6NQTVICRHJSDUKF7Q6RM5N4CCMQ5SJPYWQ5AFV73Y4YLUKTOYA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.427259020 (381h4m14.298342055s) [noise=true , address=145.79.227.49:4337 , piece_id=ENA2BTOYC3GPDXP2Z54TPCAWUDYFFD6WRJAKQCZPR2GMDQHJAQPQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: <nil> Dec 21 12:33:14.427283867 (381h4m14.298366912s) [seg_pos={0 0}, upload=334427]: Segment reached optimal threshold of 13 pieces. Dec 21 12:33:14.427589459 (381h4m14.298672494s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 21 12:33:14.427598365 (381h4m14.298681401s) [noise=true , address=145.79.227.16:4911 , piece_id=7ABEAA6K2VYVOQHCC43U6ACCKGDSSTZIBHTCTBXTW4UDRSNEFPWQ, seg_pos={0 0}, upload=334427]: Long tail cancellation due to optimal threshold Dec 21 12:33:14.427977878 (381h4m14.299060923s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 21 12:33:14.427987356 (381h4m14.299070391s) [noise=true , address=145.79.227.17:4972 , piece_id=FBJP7OUDAKV7ZQGY5HAU2FR2CI5INMQXT2LRMWZ5V6RRC4XDNKCA, seg_pos={0 0}, upload=334427]: Long tail cancellation due to optimal threshold Dec 21 12:33:14.428082487 (381h4m14.299165522s) [seg_pos={0 0}, upload=334427]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:33:14.428189882 (381h4m14.299272917s) [upload=334427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:33:14.428196234 (381h4m14.299279279s) [upload=334427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:33:14.725380605 (381h4m14.59646365s) [upload=334427]: Done uploading segments... Dec 21 12:33:14.725423627 (381h4m14.596506672s) [upload=334427]: Upload finished. err: <nil> Duration: 30.00271328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..................XX.X..X.X..X.XXX.X.X..X.........X.X.X.....] duration: 5.689979917s trace: https://jaeger.dev.storj.io/trace/2a2bce88df87643f FAIL (2025-12-21 12:32:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: 3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA) context deadline exceeded ======LOGS====== Dec 21 12:29:10.908608899 (381h0m10.779691934s) [upload=334370]: Starting upload Dec 21 12:29:10.908670967 (381h0m10.779754003s) [upload=334370]: Done starting upload Dec 21 12:29:10.908830912 (381h0m10.779913958s) [upload=334370]: Uploading segments... Dec 21 12:29:10.908909802 (381h0m10.779992837s) [upload=334370]: Got next segment. Inline: false Dec 21 12:29:10.908955820 (381h0m10.780038865s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:29:10.908962843 (381h0m10.780045888s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:29:11.169371890 (381h0m11.040454935s) [seg_pos={0 0}, upload=334370]: Begin upload segment... Dec 21 12:29:11.169471480 (381h0m11.040554515s) [seg_pos={0 0}, upload=334370]: Done begin upload segment. Dec 21 12:29:11.169516675 (381h0m11.040599721s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.169528698 (381h0m11.040611733s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170455884 (381h0m11.041538929s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170898477 (381h0m11.041981532s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171242572 (381h0m11.042325617s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171364183 (381h0m11.042447228s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171654255 (381h0m11.0427373s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172030953 (381h0m11.043113988s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172603013 (381h0m11.043686048s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172662265 (381h0m11.04374531s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173150905 (381h0m11.04423395s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173750909 (381h0m11.044833944s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174120371 (381h0m11.045203407s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174569918 (381h0m11.045652953s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174898073 (381h0m11.045981118s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.175299387 (381h0m11.046382432s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:13.030338713 (381h0m12.901421758s) [upload=334370]: Next returned nil segment Dec 21 12:29:13.030355225 (381h0m12.90143827s) [upload=334370]: Waiting for error group managing segments... Dec 21 12:29:13.333675110 (381h0m13.204758145s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.348169290 (381h0m13.219252335s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.355626129 (381h0m13.226709164s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.382155103 (381h0m13.253238138s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.402329478 (381h0m13.273412513s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.491273721 (381h0m13.362356756s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.588606817 (381h0m13.459689862s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.607294722 (381h0m13.478377757s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.729844575 (381h0m13.60092762s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.032823782 (381h0m13.903906817s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.110954245 (381h0m13.98203728s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.125112333 (381h0m13.996195368s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372107058 (381h0m14.243190093s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372139399 (381h0m14.243222434s) [seg_pos={0 0}, upload=334370]: Segment reached optimal threshold of 13 pieces. Dec 21 12:29:14.372368705 (381h0m14.24345174s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372377703 (381h0m14.243460738s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372487823 (381h0m14.243570858s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372503783 (381h0m14.243586828s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372621287 (381h0m14.243704322s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection Dec 21 12:29:14.372628811 (381h0m14.243711856s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372645713 (381h0m14.243728758s) [seg_pos={0 0}, upload=334370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:29:14.372741535 (381h0m14.243824571s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:29:14.372746986 (381h0m14.243830021s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:29:14.996259845 (381h0m14.86734289s) [upload=334370]: Done uploading segments... Dec 21 12:29:14.996279472 (381h0m14.867362507s) [upload=334370]: Upload finished. err: <nil> Duration: 5.69001308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...................XX.X..X.X..X.XXX.X.X..X.........X.X.X....] duration: 14.672759764s trace: https://jaeger.dev.storj.io/trace/2a2bce88df87643f FAIL (2025-12-21 12:32:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: 3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA) context deadline exceeded ======LOGS====== Dec 21 12:29:10.908608899 (381h0m10.779691934s) [upload=334370]: Starting upload Dec 21 12:29:10.908670967 (381h0m10.779754003s) [upload=334370]: Done starting upload Dec 21 12:29:10.908830912 (381h0m10.779913958s) [upload=334370]: Uploading segments... Dec 21 12:29:10.908909802 (381h0m10.779992837s) [upload=334370]: Got next segment. Inline: false Dec 21 12:29:10.908955820 (381h0m10.780038865s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:29:10.908962843 (381h0m10.780045888s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:29:11.169371890 (381h0m11.040454935s) [seg_pos={0 0}, upload=334370]: Begin upload segment... Dec 21 12:29:11.169471480 (381h0m11.040554515s) [seg_pos={0 0}, upload=334370]: Done begin upload segment. Dec 21 12:29:11.169516675 (381h0m11.040599721s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.169528698 (381h0m11.040611733s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170455884 (381h0m11.041538929s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170898477 (381h0m11.041981532s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171242572 (381h0m11.042325617s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171364183 (381h0m11.042447228s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171654255 (381h0m11.0427373s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172030953 (381h0m11.043113988s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172603013 (381h0m11.043686048s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172662265 (381h0m11.04374531s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173150905 (381h0m11.04423395s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173750909 (381h0m11.044833944s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174120371 (381h0m11.045203407s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174569918 (381h0m11.045652953s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174898073 (381h0m11.045981118s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.175299387 (381h0m11.046382432s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:13.030338713 (381h0m12.901421758s) [upload=334370]: Next returned nil segment Dec 21 12:29:13.030355225 (381h0m12.90143827s) [upload=334370]: Waiting for error group managing segments... Dec 21 12:29:13.333675110 (381h0m13.204758145s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.348169290 (381h0m13.219252335s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.355626129 (381h0m13.226709164s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.382155103 (381h0m13.253238138s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.402329478 (381h0m13.273412513s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.491273721 (381h0m13.362356756s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.588606817 (381h0m13.459689862s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.607294722 (381h0m13.478377757s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.729844575 (381h0m13.60092762s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.032823782 (381h0m13.903906817s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.110954245 (381h0m13.98203728s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.125112333 (381h0m13.996195368s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372107058 (381h0m14.243190093s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372139399 (381h0m14.243222434s) [seg_pos={0 0}, upload=334370]: Segment reached optimal threshold of 13 pieces. Dec 21 12:29:14.372368705 (381h0m14.24345174s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372377703 (381h0m14.243460738s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372487823 (381h0m14.243570858s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372503783 (381h0m14.243586828s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372621287 (381h0m14.243704322s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection Dec 21 12:29:14.372628811 (381h0m14.243711856s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372645713 (381h0m14.243728758s) [seg_pos={0 0}, upload=334370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:29:14.372741535 (381h0m14.243824571s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:29:14.372746986 (381h0m14.243830021s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:29:14.996259845 (381h0m14.86734289s) [upload=334370]: Done uploading segments... Dec 21 12:29:14.996279472 (381h0m14.867362507s) [upload=334370]: Upload finished. err: <nil> Duration: 14.67278935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....................XX.X..X.X..X.XXX.X.X..X.........X.X.X...] duration: 5.708739385s trace: https://jaeger.dev.storj.io/trace/2a2bce88df87643f FAIL (2025-12-21 12:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: 3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA) context deadline exceeded ======LOGS====== Dec 21 12:29:10.908608899 (381h0m10.779691934s) [upload=334370]: Starting upload Dec 21 12:29:10.908670967 (381h0m10.779754003s) [upload=334370]: Done starting upload Dec 21 12:29:10.908830912 (381h0m10.779913958s) [upload=334370]: Uploading segments... Dec 21 12:29:10.908909802 (381h0m10.779992837s) [upload=334370]: Got next segment. Inline: false Dec 21 12:29:10.908955820 (381h0m10.780038865s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:29:10.908962843 (381h0m10.780045888s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:29:11.169371890 (381h0m11.040454935s) [seg_pos={0 0}, upload=334370]: Begin upload segment... Dec 21 12:29:11.169471480 (381h0m11.040554515s) [seg_pos={0 0}, upload=334370]: Done begin upload segment. Dec 21 12:29:11.169516675 (381h0m11.040599721s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.169528698 (381h0m11.040611733s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170455884 (381h0m11.041538929s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170898477 (381h0m11.041981532s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171242572 (381h0m11.042325617s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171364183 (381h0m11.042447228s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171654255 (381h0m11.0427373s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172030953 (381h0m11.043113988s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172603013 (381h0m11.043686048s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172662265 (381h0m11.04374531s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173150905 (381h0m11.04423395s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173750909 (381h0m11.044833944s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174120371 (381h0m11.045203407s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174569918 (381h0m11.045652953s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174898073 (381h0m11.045981118s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.175299387 (381h0m11.046382432s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:13.030338713 (381h0m12.901421758s) [upload=334370]: Next returned nil segment Dec 21 12:29:13.030355225 (381h0m12.90143827s) [upload=334370]: Waiting for error group managing segments... Dec 21 12:29:13.333675110 (381h0m13.204758145s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.348169290 (381h0m13.219252335s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.355626129 (381h0m13.226709164s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.382155103 (381h0m13.253238138s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.402329478 (381h0m13.273412513s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.491273721 (381h0m13.362356756s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.588606817 (381h0m13.459689862s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.607294722 (381h0m13.478377757s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.729844575 (381h0m13.60092762s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.032823782 (381h0m13.903906817s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.110954245 (381h0m13.98203728s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.125112333 (381h0m13.996195368s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372107058 (381h0m14.243190093s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372139399 (381h0m14.243222434s) [seg_pos={0 0}, upload=334370]: Segment reached optimal threshold of 13 pieces. Dec 21 12:29:14.372368705 (381h0m14.24345174s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372377703 (381h0m14.243460738s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372487823 (381h0m14.243570858s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372503783 (381h0m14.243586828s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372621287 (381h0m14.243704322s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection Dec 21 12:29:14.372628811 (381h0m14.243711856s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372645713 (381h0m14.243728758s) [seg_pos={0 0}, upload=334370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:29:14.372741535 (381h0m14.243824571s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:29:14.372746986 (381h0m14.243830021s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:29:14.996259845 (381h0m14.86734289s) [upload=334370]: Done uploading segments... Dec 21 12:29:14.996279472 (381h0m14.867362507s) [upload=334370]: Upload finished. err: <nil> Duration: 5.708758621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....................XX.X..X.X..X.XXX.X.X..X.........X.X.X..] duration: 6.755202044s trace: https://jaeger.dev.storj.io/trace/2a2bce88df87643f FAIL (2025-12-21 12:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: 3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA) context deadline exceeded ======LOGS====== Dec 21 12:29:10.908608899 (381h0m10.779691934s) [upload=334370]: Starting upload Dec 21 12:29:10.908670967 (381h0m10.779754003s) [upload=334370]: Done starting upload Dec 21 12:29:10.908830912 (381h0m10.779913958s) [upload=334370]: Uploading segments... Dec 21 12:29:10.908909802 (381h0m10.779992837s) [upload=334370]: Got next segment. Inline: false Dec 21 12:29:10.908955820 (381h0m10.780038865s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:29:10.908962843 (381h0m10.780045888s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:29:11.169371890 (381h0m11.040454935s) [seg_pos={0 0}, upload=334370]: Begin upload segment... Dec 21 12:29:11.169471480 (381h0m11.040554515s) [seg_pos={0 0}, upload=334370]: Done begin upload segment. Dec 21 12:29:11.169516675 (381h0m11.040599721s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.169528698 (381h0m11.040611733s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170455884 (381h0m11.041538929s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170898477 (381h0m11.041981532s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171242572 (381h0m11.042325617s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171364183 (381h0m11.042447228s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171654255 (381h0m11.0427373s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172030953 (381h0m11.043113988s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172603013 (381h0m11.043686048s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172662265 (381h0m11.04374531s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173150905 (381h0m11.04423395s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173750909 (381h0m11.044833944s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174120371 (381h0m11.045203407s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174569918 (381h0m11.045652953s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174898073 (381h0m11.045981118s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.175299387 (381h0m11.046382432s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:13.030338713 (381h0m12.901421758s) [upload=334370]: Next returned nil segment Dec 21 12:29:13.030355225 (381h0m12.90143827s) [upload=334370]: Waiting for error group managing segments... Dec 21 12:29:13.333675110 (381h0m13.204758145s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.348169290 (381h0m13.219252335s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.355626129 (381h0m13.226709164s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.382155103 (381h0m13.253238138s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.402329478 (381h0m13.273412513s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.491273721 (381h0m13.362356756s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.588606817 (381h0m13.459689862s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.607294722 (381h0m13.478377757s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.729844575 (381h0m13.60092762s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.032823782 (381h0m13.903906817s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.110954245 (381h0m13.98203728s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.125112333 (381h0m13.996195368s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372107058 (381h0m14.243190093s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372139399 (381h0m14.243222434s) [seg_pos={0 0}, upload=334370]: Segment reached optimal threshold of 13 pieces. Dec 21 12:29:14.372368705 (381h0m14.24345174s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372377703 (381h0m14.243460738s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372487823 (381h0m14.243570858s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372503783 (381h0m14.243586828s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372621287 (381h0m14.243704322s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection Dec 21 12:29:14.372628811 (381h0m14.243711856s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372645713 (381h0m14.243728758s) [seg_pos={0 0}, upload=334370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:29:14.372741535 (381h0m14.243824571s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:29:14.372746986 (381h0m14.243830021s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:29:14.996259845 (381h0m14.86734289s) [upload=334370]: Done uploading segments... Dec 21 12:29:14.996279472 (381h0m14.867362507s) [upload=334370]: Upload finished. err: <nil> Duration: 6.755234316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......................XX.X..X.X..X.XXX.X.X..X.........X.X.X.] duration: 5.692268884s trace: https://jaeger.dev.storj.io/trace/2a2bce88df87643f FAIL (2025-12-21 12:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: 3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA) context deadline exceeded ======LOGS====== Dec 21 12:29:10.908608899 (381h0m10.779691934s) [upload=334370]: Starting upload Dec 21 12:29:10.908670967 (381h0m10.779754003s) [upload=334370]: Done starting upload Dec 21 12:29:10.908830912 (381h0m10.779913958s) [upload=334370]: Uploading segments... Dec 21 12:29:10.908909802 (381h0m10.779992837s) [upload=334370]: Got next segment. Inline: false Dec 21 12:29:10.908955820 (381h0m10.780038865s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:29:10.908962843 (381h0m10.780045888s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:29:11.169371890 (381h0m11.040454935s) [seg_pos={0 0}, upload=334370]: Begin upload segment... Dec 21 12:29:11.169471480 (381h0m11.040554515s) [seg_pos={0 0}, upload=334370]: Done begin upload segment. Dec 21 12:29:11.169516675 (381h0m11.040599721s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.169528698 (381h0m11.040611733s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170455884 (381h0m11.041538929s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170898477 (381h0m11.041981532s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171242572 (381h0m11.042325617s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171364183 (381h0m11.042447228s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171654255 (381h0m11.0427373s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172030953 (381h0m11.043113988s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172603013 (381h0m11.043686048s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172662265 (381h0m11.04374531s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173150905 (381h0m11.04423395s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173750909 (381h0m11.044833944s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174120371 (381h0m11.045203407s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174569918 (381h0m11.045652953s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174898073 (381h0m11.045981118s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.175299387 (381h0m11.046382432s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:13.030338713 (381h0m12.901421758s) [upload=334370]: Next returned nil segment Dec 21 12:29:13.030355225 (381h0m12.90143827s) [upload=334370]: Waiting for error group managing segments... Dec 21 12:29:13.333675110 (381h0m13.204758145s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.348169290 (381h0m13.219252335s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.355626129 (381h0m13.226709164s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.382155103 (381h0m13.253238138s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.402329478 (381h0m13.273412513s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.491273721 (381h0m13.362356756s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.588606817 (381h0m13.459689862s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.607294722 (381h0m13.478377757s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.729844575 (381h0m13.60092762s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.032823782 (381h0m13.903906817s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.110954245 (381h0m13.98203728s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.125112333 (381h0m13.996195368s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372107058 (381h0m14.243190093s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372139399 (381h0m14.243222434s) [seg_pos={0 0}, upload=334370]: Segment reached optimal threshold of 13 pieces. Dec 21 12:29:14.372368705 (381h0m14.24345174s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372377703 (381h0m14.243460738s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372487823 (381h0m14.243570858s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372503783 (381h0m14.243586828s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372621287 (381h0m14.243704322s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection Dec 21 12:29:14.372628811 (381h0m14.243711856s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372645713 (381h0m14.243728758s) [seg_pos={0 0}, upload=334370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:29:14.372741535 (381h0m14.243824571s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:29:14.372746986 (381h0m14.243830021s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:29:14.996259845 (381h0m14.86734289s) [upload=334370]: Done uploading segments... Dec 21 12:29:14.996279472 (381h0m14.867362507s) [upload=334370]: Upload finished. err: <nil> Duration: 5.692295204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......................XX.X..X.X..X.XXX.X.X..X.........X.X.X] duration: 30.003130479s trace: https://jaeger.dev.storj.io/trace/2a2bce88df87643f FAIL (2025-12-21 12:29:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA) context deadline exceeded; piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: 3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA) context deadline exceeded ======LOGS====== Dec 21 12:29:10.908608899 (381h0m10.779691934s) [upload=334370]: Starting upload Dec 21 12:29:10.908670967 (381h0m10.779754003s) [upload=334370]: Done starting upload Dec 21 12:29:10.908830912 (381h0m10.779913958s) [upload=334370]: Uploading segments... Dec 21 12:29:10.908909802 (381h0m10.779992837s) [upload=334370]: Got next segment. Inline: false Dec 21 12:29:10.908955820 (381h0m10.780038865s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:29:10.908962843 (381h0m10.780045888s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:29:11.169371890 (381h0m11.040454935s) [seg_pos={0 0}, upload=334370]: Begin upload segment... Dec 21 12:29:11.169471480 (381h0m11.040554515s) [seg_pos={0 0}, upload=334370]: Done begin upload segment. Dec 21 12:29:11.169516675 (381h0m11.040599721s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.169528698 (381h0m11.040611733s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170455884 (381h0m11.041538929s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.170898477 (381h0m11.041981532s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171242572 (381h0m11.042325617s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171364183 (381h0m11.042447228s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.171654255 (381h0m11.0427373s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172030953 (381h0m11.043113988s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172603013 (381h0m11.043686048s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.172662265 (381h0m11.04374531s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173150905 (381h0m11.04423395s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.173750909 (381h0m11.044833944s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174120371 (381h0m11.045203407s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174569918 (381h0m11.045652953s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.174898073 (381h0m11.045981118s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:11.175299387 (381h0m11.046382432s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Uploading piece... Dec 21 12:29:13.030338713 (381h0m12.901421758s) [upload=334370]: Next returned nil segment Dec 21 12:29:13.030355225 (381h0m12.90143827s) [upload=334370]: Waiting for error group managing segments... Dec 21 12:29:13.333675110 (381h0m13.204758145s) [noise=true , address=145.79.227.48:4301 , piece_id=ADHGNMDAMGUUSHQPQZHNRR44PZKKPKSX255ZHGTOUO66ZHFJQJHQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.348169290 (381h0m13.219252335s) [noise=true , address=145.79.227.34:4907 , piece_id=3AMFT3GZJGSGX2IHO4MX5CBBP52BRDEFJPYUDRG7PJ2YB7YEEWEA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.355626129 (381h0m13.226709164s) [noise=true , address=145.79.227.52:4447 , piece_id=3OMBC3645DVW2RXF4ZK7AS5JO5AO4777GH5B5QQH25MSMAOJAWBA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.382155103 (381h0m13.253238138s) [noise=true , address=145.79.227.47:4266 , piece_id=EEOEMBFHZCOEUBNV6UDEG2KTOGJEYZLNQ46LDOZTU32KZ4W5EBLQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.402329478 (381h0m13.273412513s) [noise=true , address=145.79.227.33:4870 , piece_id=GVE57FAUO3EZPHR3K757G4OT4T6HOTFDJGKXSGRXH4TVG7P7HWGA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.491273721 (381h0m13.362356756s) [noise=true , address=45.91.116.9:4264 , piece_id=RUVO6WI5M7MOQ3RWUTQUTXET6BEOGY75PPPSXMIBWUDRYU4IBF3Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.588606817 (381h0m13.459689862s) [noise=true , address=45.91.116.14:4189 , piece_id=NN3AFQCM6ASH6KNLQH7KN3FNLRB7GI6JWAH5QFK3IVD37IRFCMSA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.607294722 (381h0m13.478377757s) [noise=true , address=45.91.116.4:4940 , piece_id=VYQJBNA3FC6ZAOTYPAE3PJFBQSE6IOEWP4UNBVXHJRA6IOM5P4WA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:13.729844575 (381h0m13.60092762s) [noise=true , address=145.79.227.17:4972 , piece_id=DMZ523JLK43ZELDINBXHLQAEKFVC35BHWTIP3PIS3ZEAPIKZRCZA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.032823782 (381h0m13.903906817s) [noise=true , address=145.79.227.49:4337 , piece_id=THWLCCLEGRWUTVXSIXAH6L2ZRV4V2FVTQTC4KNXUAXVLXE6SPU7Q, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.110954245 (381h0m13.98203728s) [noise=true , address=145.79.227.41:4086 , piece_id=BAZXKFGNI7SHX4753QGBZ372ECNMC5NIWFN5BX3ZBFYMP2ZOBIXA, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.125112333 (381h0m13.996195368s) [noise=true , address=145.79.227.32:4835 , piece_id=M2YEC3HYMEZIWAGUS37FCTH7ZFAFISXU34ALTB3UNRC3V5DMU47A, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372107058 (381h0m14.243190093s) [noise=true , address=145.79.227.29:4943 , piece_id=KMHT324UUUPVCKPJVJADEABOL7KSTCQYAWVSILER4M5XL2L6YDGQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: <nil> Dec 21 12:29:14.372139399 (381h0m14.243222434s) [seg_pos={0 0}, upload=334370]: Segment reached optimal threshold of 13 pieces. Dec 21 12:29:14.372368705 (381h0m14.24345174s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372377703 (381h0m14.243460738s) [noise=true , address=145.79.227.40:4049 , piece_id=OMP4HABXQOUPFJIOACW46MMUWGJWV3TJB2ATBBB55H2XNAIWMMWQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372487823 (381h0m14.243570858s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 21 12:29:14.372503783 (381h0m14.243586828s) [noise=true , address=145.79.227.2:4115 , piece_id=HSIYS7OZCTCKT4Q7TGFNSFSAR25JCKBLFKVZFOBMVZBZ25LTNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372621287 (381h0m14.243704322s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50768->145.79.227.10:4403: use of closed network connection Dec 21 12:29:14.372628811 (381h0m14.243711856s) [noise=true , address=145.79.227.10:4403 , piece_id=VCOTWUPKG3XRZID7HKAY6LDRELA35EM3ZWYYQWADFAUDBN5BNCJQ, seg_pos={0 0}, upload=334370]: Long tail cancellation due to optimal threshold Dec 21 12:29:14.372645713 (381h0m14.243728758s) [seg_pos={0 0}, upload=334370]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:29:14.372741535 (381h0m14.243824571s) [upload=334370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:29:14.372746986 (381h0m14.243830021s) [upload=334370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:29:14.996259845 (381h0m14.86734289s) [upload=334370]: Done uploading segments... Dec 21 12:29:14.996279472 (381h0m14.867362507s) [upload=334370]: Upload finished. err: <nil> Duration: 30.003153503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................XX.X..X.X..X.XXX.X.X..X.........X.X.] duration: 5.643083056s trace: https://jaeger.dev.storj.io/trace/730bd334780b3bfd FAIL (2025-12-21 12:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: R5YOOEIWH2ZI7MPVVETTBXPDVS5CXAT2YBZ36VIDVXUBNOWUFOUA) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: ALAVIVRK2OEFAHYJ4QIKEQ4NWYZI5ZIIVLMK7GSJYOQTVK2R72EQ) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: PVKTFAUSZVJTGC3SWC5RHNSB3OLGPIFXK6RPP27BCZKCFCFK7YZA) context deadline exceeded; piecestore: (Node ID: 12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, Piece ID: CYTIVRBL53LNPJ72NOHFQD242ME7EXFWU2NMPOBZWGMSNKCZAS5A) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: QO36WGV7WPTX3ZR5RXYUT6DDZYCXF6XCSIWCINOBBRSG5443BFLA) context deadline exceeded ======LOGS====== d=334349]: Uploading segments... Dec 21 12:27:39.039922661 (380h58m38.911005696s) [upload=334349]: Got next segment. Inline: false Dec 21 12:27:39.039951566 (380h58m38.911034601s) [upload=334349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:27:39.039958258 (380h58m38.911041293s) [upload=334349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:27:39.546135944 (380h58m39.417218989s) [seg_pos={0 0}, upload=334349]: Begin upload segment... Dec 21 12:27:39.546190278 (380h58m39.417273323s) [seg_pos={0 0}, upload=334349]: Done begin upload segment. Dec 21 12:27:39.546224704 (380h58m39.417307749s) [noise=true , address=145.79.227.36:4517 , piece_id=PVKTFAUSZVJTGC3SWC5RHNSB3OLGPIFXK6RPP27BCZKCFCFK7YZA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.546254951 (380h58m39.417337986s) [noise=true , address=145.79.227.20:4439 , piece_id=G6UALNJ7ISVIBB7ZIY63RWQ7DOL54QB6WNOQLM6E77WXWGKACQGQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.546371513 (380h58m39.417454558s) [noise=true , address=145.79.227.40:4050 , piece_id=CYTIVRBL53LNPJ72NOHFQD242ME7EXFWU2NMPOBZWGMSNKCZAS5A, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.546962599 (380h58m39.418045644s) [noise=true , address=45.91.116.9:4262 , piece_id=HPRUBGKFQCJIUTRMPFGQKSYW6FPSI6WHX7OHFT2IVHV5RVYWYFPQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.547304580 (380h58m39.418387625s) [noise=true , address=145.79.227.14:4839 , piece_id=XJKMNE3PHBV2XUSGX3GUJEWABW2WZB77SGXCHELULMMV2GJ2IVEA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.547641391 (380h58m39.418724426s) [noise=true , address=145.79.227.15:4875 , piece_id=L7VBEME77KXSL7VAUPC4VNWAQABMUU4FKRGPTBEY5UVCHNCBNMJA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.547984865 (380h58m39.41906791s) [noise=true , address=145.79.227.28:4726 , piece_id=K5JT244HN4R7GZOBP3KODN4BLKFBZOZGO3MDSBU42NQ3RCX3HTFA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.548347846 (380h58m39.419430882s) [noise=true , address=77.90.31.17:4770 , piece_id=TKC2KQIXO4VEY7TQAVUNCVOACI27DTTRB4GSSVTGNVRULR7O4IIQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.548857206 (380h58m39.419940241s) [noise=true , address=145.79.227.34:4905 , piece_id=MO6ZE4SFTFSWHK37MTLXLIOB25NZ7WSS6C7IX366OLYJ5ANG5JIQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.550875319 (380h58m39.421958364s) [noise=true , address=145.79.227.39:4013 , piece_id=EOE6ICL7RFQNLIH4D223FGXOADJDD6RNAL6WK2YKL6ORC2QNKMFQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.551991594 (380h58m39.423074629s) [noise=true , address=145.79.227.43:4157 , piece_id=QO36WGV7WPTX3ZR5RXYUT6DDZYCXF6XCSIWCINOBBRSG5443BFLA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.554746030 (380h58m39.425829065s) [noise=true , address=45.91.116.8:4046 , piece_id=YHREOU54WTEIPWSBF2P2N3RALJJNGAEQEWHRXU7HJRVK2S5STXPA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.556031938 (380h58m39.427114973s) [noise=true , address=145.79.227.42:4123 , piece_id=R5YOOEIWH2ZI7MPVVETTBXPDVS5CXAT2YBZ36VIDVXUBNOWUFOUA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.557279593 (380h58m39.428362629s) [noise=true , address=145.79.227.37:4567 , piece_id=44QCNL7ZZHZ6KC2YT7EBYUCWCM4CRQN2V5JXQLCWUVTI7CP332YQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.558653520 (380h58m39.429736555s) [noise=true , address=145.79.227.38:4978 , piece_id=ALAVIVRK2OEFAHYJ4QIKEQ4NWYZI5ZIIVLMK7GSJYOQTVK2R72EQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.560013760 (380h58m39.431096795s) [noise=true , address=77.90.31.11:4556 , piece_id=PDWFQ7GZYRGSNFHRRPEJIBMWGCKCUXKD4HPAVK7VSNEIUJXWNEUA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:41.379788827 (380h58m41.250871872s) [upload=334349]: Next returned nil segment Dec 21 12:27:41.379806611 (380h58m41.250889656s) [upload=334349]: Waiting for error group managing segments... Dec 21 12:27:41.780612354 (380h58m41.651695399s) [noise=true , address=77.90.31.11:4556 , piece_id=PDWFQ7GZYRGSNFHRRPEJIBMWGCKCUXKD4HPAVK7VSNEIUJXWNEUA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.832903264 (380h58m41.703986299s) [noise=true , address=145.79.227.43:4157 , piece_id=QO36WGV7WPTX3ZR5RXYUT6DDZYCXF6XCSIWCINOBBRSG5443BFLA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.850414428 (380h58m41.721497473s) [noise=true , address=45.91.116.8:4046 , piece_id=YHREOU54WTEIPWSBF2P2N3RALJJNGAEQEWHRXU7HJRVK2S5STXPA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.862638024 (380h58m41.733721059s) [noise=true , address=77.90.31.17:4770 , piece_id=TKC2KQIXO4VEY7TQAVUNCVOACI27DTTRB4GSSVTGNVRULR7O4IIQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.862852221 (380h58m41.733935256s) [noise=true , address=45.91.116.9:4262 , piece_id=HPRUBGKFQCJIUTRMPFGQKSYW6FPSI6WHX7OHFT2IVHV5RVYWYFPQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.413202959 (380h58m42.284285994s) [noise=true , address=145.79.227.40:4050 , piece_id=CYTIVRBL53LNPJ72NOHFQD242ME7EXFWU2NMPOBZWGMSNKCZAS5A, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.583552590 (380h58m42.454635635s) [noise=true , address=145.79.227.36:4517 , piece_id=PVKTFAUSZVJTGC3SWC5RHNSB3OLGPIFXK6RPP27BCZKCFCFK7YZA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.808082187 (380h58m42.679165232s) [noise=true , address=145.79.227.42:4123 , piece_id=R5YOOEIWH2ZI7MPVVETTBXPDVS5CXAT2YBZ36VIDVXUBNOWUFOUA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.866773675 (380h58m42.73785671s) [noise=true , address=145.79.227.14:4839 , piece_id=XJKMNE3PHBV2XUSGX3GUJEWABW2WZB77SGXCHELULMMV2GJ2IVEA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:43.222567068 (380h58m43.093650113s) [noise=true , address=145.79.227.37:4567 , piece_id=44QCNL7ZZHZ6KC2YT7EBYUCWCM4CRQN2V5JXQLCWUVTI7CP332YQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:43.284370000 (380h58m43.155453045s) [noise=true , address=145.79.227.38:4978 , piece_id=ALAVIVRK2OEFAHYJ4QIKEQ4NWYZI5ZIIVLMK7GSJYOQTVK2R72EQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:43.352548983 (380h58m43.223632018s) [noise=true , address=145.79.227.15:4875 , piece_id=L7VBEME77KXSL7VAUPC4VNWAQABMUU4FKRGPTBEY5UVCHNCBNMJA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:44.390846803 (380h58m44.261929838s) [noise=true , address=145.79.227.28:4726 , piece_id=K5JT244HN4R7GZOBP3KODN4BLKFBZOZGO3MDSBU42NQ3RCX3HTFA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:44.390866231 (380h58m44.261949276s) [seg_pos={0 0}, upload=334349]: Segment reached optimal threshold of 13 pieces. Dec 21 12:27:44.391096118 (380h58m44.262179153s) [noise=true , address=145.79.227.34:4905 , piece_id=MO6ZE4SFTFSWHK37MTLXLIOB25NZ7WSS6C7IX366OLYJ5ANG5JIQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: read tcp 188.166.225.170:33544->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:27:44.391102119 (380h58m44.262185154s) [noise=true , address=145.79.227.34:4905 , piece_id=MO6ZE4SFTFSWHK37MTLXLIOB25NZ7WSS6C7IX366OLYJ5ANG5JIQ, seg_pos={0 0}, upload=334349]: Long tail cancellation due to optimal threshold Dec 21 12:27:44.391391150 (380h58m44.262474195s) [noise=true , address=145.79.227.20:4439 , piece_id=G6UALNJ7ISVIBB7ZIY63RWQ7DOL54QB6WNOQLM6E77WXWGKACQGQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: write tcp 188.166.225.170:60238->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:60238->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60238->145.79.227.20:4439: use of closed network connection Dec 21 12:27:44.391398514 (380h58m44.262481549s) [noise=true , address=145.79.227.20:4439 , piece_id=G6UALNJ7ISVIBB7ZIY63RWQ7DOL54QB6WNOQLM6E77WXWGKACQGQ, seg_pos={0 0}, upload=334349]: Long tail cancellation due to optimal threshold Dec 21 12:27:44.391543640 (380h58m44.262626675s) [noise=true , address=145.79.227.39:4013 , piece_id=EOE6ICL7RFQNLIH4D223FGXOADJDD6RNAL6WK2YKL6ORC2QNKMFQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:34860->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:27:44.391549221 (380h58m44.262632256s) [noise=true , address=145.79.227.39:4013 , piece_id=EOE6ICL7RFQNLIH4D223FGXOADJDD6RNAL6WK2YKL6ORC2QNKMFQ, seg_pos={0 0}, upload=334349]: Long tail cancellation due to optimal threshold Dec 21 12:27:44.391561946 (380h58m44.262644981s) [seg_pos={0 0}, upload=334349]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:27:44.391623422 (380h58m44.262706457s) [upload=334349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:27:44.391628052 (380h58m44.262711097s) [upload=334349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:27:44.708357059 (380h58m44.579440104s) [upload=334349]: Done uploading segments... Dec 21 12:27:44.708380202 (380h58m44.579463247s) [upload=334349]: Upload finished. err: <nil> Duration: 5.643112441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................XX.X..X.X..X.XXX.X.X..X.........X.X] duration: 30.00456348s trace: https://jaeger.dev.storj.io/trace/730bd334780b3bfd FAIL (2025-12-21 12:28:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: R5YOOEIWH2ZI7MPVVETTBXPDVS5CXAT2YBZ36VIDVXUBNOWUFOUA) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: ALAVIVRK2OEFAHYJ4QIKEQ4NWYZI5ZIIVLMK7GSJYOQTVK2R72EQ) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: PVKTFAUSZVJTGC3SWC5RHNSB3OLGPIFXK6RPP27BCZKCFCFK7YZA) context deadline exceeded; piecestore: (Node ID: 12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, Piece ID: CYTIVRBL53LNPJ72NOHFQD242ME7EXFWU2NMPOBZWGMSNKCZAS5A) context deadline exceeded; piecestore: (Node ID: 1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, Piece ID: QO36WGV7WPTX3ZR5RXYUT6DDZYCXF6XCSIWCINOBBRSG5443BFLA) context deadline exceeded ======LOGS====== d=334349]: Uploading segments... Dec 21 12:27:39.039922661 (380h58m38.911005696s) [upload=334349]: Got next segment. Inline: false Dec 21 12:27:39.039951566 (380h58m38.911034601s) [upload=334349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:27:39.039958258 (380h58m38.911041293s) [upload=334349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:27:39.546135944 (380h58m39.417218989s) [seg_pos={0 0}, upload=334349]: Begin upload segment... Dec 21 12:27:39.546190278 (380h58m39.417273323s) [seg_pos={0 0}, upload=334349]: Done begin upload segment. Dec 21 12:27:39.546224704 (380h58m39.417307749s) [noise=true , address=145.79.227.36:4517 , piece_id=PVKTFAUSZVJTGC3SWC5RHNSB3OLGPIFXK6RPP27BCZKCFCFK7YZA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.546254951 (380h58m39.417337986s) [noise=true , address=145.79.227.20:4439 , piece_id=G6UALNJ7ISVIBB7ZIY63RWQ7DOL54QB6WNOQLM6E77WXWGKACQGQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.546371513 (380h58m39.417454558s) [noise=true , address=145.79.227.40:4050 , piece_id=CYTIVRBL53LNPJ72NOHFQD242ME7EXFWU2NMPOBZWGMSNKCZAS5A, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.546962599 (380h58m39.418045644s) [noise=true , address=45.91.116.9:4262 , piece_id=HPRUBGKFQCJIUTRMPFGQKSYW6FPSI6WHX7OHFT2IVHV5RVYWYFPQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.547304580 (380h58m39.418387625s) [noise=true , address=145.79.227.14:4839 , piece_id=XJKMNE3PHBV2XUSGX3GUJEWABW2WZB77SGXCHELULMMV2GJ2IVEA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.547641391 (380h58m39.418724426s) [noise=true , address=145.79.227.15:4875 , piece_id=L7VBEME77KXSL7VAUPC4VNWAQABMUU4FKRGPTBEY5UVCHNCBNMJA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.547984865 (380h58m39.41906791s) [noise=true , address=145.79.227.28:4726 , piece_id=K5JT244HN4R7GZOBP3KODN4BLKFBZOZGO3MDSBU42NQ3RCX3HTFA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.548347846 (380h58m39.419430882s) [noise=true , address=77.90.31.17:4770 , piece_id=TKC2KQIXO4VEY7TQAVUNCVOACI27DTTRB4GSSVTGNVRULR7O4IIQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.548857206 (380h58m39.419940241s) [noise=true , address=145.79.227.34:4905 , piece_id=MO6ZE4SFTFSWHK37MTLXLIOB25NZ7WSS6C7IX366OLYJ5ANG5JIQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.550875319 (380h58m39.421958364s) [noise=true , address=145.79.227.39:4013 , piece_id=EOE6ICL7RFQNLIH4D223FGXOADJDD6RNAL6WK2YKL6ORC2QNKMFQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.551991594 (380h58m39.423074629s) [noise=true , address=145.79.227.43:4157 , piece_id=QO36WGV7WPTX3ZR5RXYUT6DDZYCXF6XCSIWCINOBBRSG5443BFLA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.554746030 (380h58m39.425829065s) [noise=true , address=45.91.116.8:4046 , piece_id=YHREOU54WTEIPWSBF2P2N3RALJJNGAEQEWHRXU7HJRVK2S5STXPA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.556031938 (380h58m39.427114973s) [noise=true , address=145.79.227.42:4123 , piece_id=R5YOOEIWH2ZI7MPVVETTBXPDVS5CXAT2YBZ36VIDVXUBNOWUFOUA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.557279593 (380h58m39.428362629s) [noise=true , address=145.79.227.37:4567 , piece_id=44QCNL7ZZHZ6KC2YT7EBYUCWCM4CRQN2V5JXQLCWUVTI7CP332YQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.558653520 (380h58m39.429736555s) [noise=true , address=145.79.227.38:4978 , piece_id=ALAVIVRK2OEFAHYJ4QIKEQ4NWYZI5ZIIVLMK7GSJYOQTVK2R72EQ, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:39.560013760 (380h58m39.431096795s) [noise=true , address=77.90.31.11:4556 , piece_id=PDWFQ7GZYRGSNFHRRPEJIBMWGCKCUXKD4HPAVK7VSNEIUJXWNEUA, seg_pos={0 0}, upload=334349]: Uploading piece... Dec 21 12:27:41.379788827 (380h58m41.250871872s) [upload=334349]: Next returned nil segment Dec 21 12:27:41.379806611 (380h58m41.250889656s) [upload=334349]: Waiting for error group managing segments... Dec 21 12:27:41.780612354 (380h58m41.651695399s) [noise=true , address=77.90.31.11:4556 , piece_id=PDWFQ7GZYRGSNFHRRPEJIBMWGCKCUXKD4HPAVK7VSNEIUJXWNEUA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.832903264 (380h58m41.703986299s) [noise=true , address=145.79.227.43:4157 , piece_id=QO36WGV7WPTX3ZR5RXYUT6DDZYCXF6XCSIWCINOBBRSG5443BFLA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.850414428 (380h58m41.721497473s) [noise=true , address=45.91.116.8:4046 , piece_id=YHREOU54WTEIPWSBF2P2N3RALJJNGAEQEWHRXU7HJRVK2S5STXPA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.862638024 (380h58m41.733721059s) [noise=true , address=77.90.31.17:4770 , piece_id=TKC2KQIXO4VEY7TQAVUNCVOACI27DTTRB4GSSVTGNVRULR7O4IIQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:41.862852221 (380h58m41.733935256s) [noise=true , address=45.91.116.9:4262 , piece_id=HPRUBGKFQCJIUTRMPFGQKSYW6FPSI6WHX7OHFT2IVHV5RVYWYFPQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.413202959 (380h58m42.284285994s) [noise=true , address=145.79.227.40:4050 , piece_id=CYTIVRBL53LNPJ72NOHFQD242ME7EXFWU2NMPOBZWGMSNKCZAS5A, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.583552590 (380h58m42.454635635s) [noise=true , address=145.79.227.36:4517 , piece_id=PVKTFAUSZVJTGC3SWC5RHNSB3OLGPIFXK6RPP27BCZKCFCFK7YZA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.808082187 (380h58m42.679165232s) [noise=true , address=145.79.227.42:4123 , piece_id=R5YOOEIWH2ZI7MPVVETTBXPDVS5CXAT2YBZ36VIDVXUBNOWUFOUA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:42.866773675 (380h58m42.73785671s) [noise=true , address=145.79.227.14:4839 , piece_id=XJKMNE3PHBV2XUSGX3GUJEWABW2WZB77SGXCHELULMMV2GJ2IVEA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:43.222567068 (380h58m43.093650113s) [noise=true , address=145.79.227.37:4567 , piece_id=44QCNL7ZZHZ6KC2YT7EBYUCWCM4CRQN2V5JXQLCWUVTI7CP332YQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:43.284370000 (380h58m43.155453045s) [noise=true , address=145.79.227.38:4978 , piece_id=ALAVIVRK2OEFAHYJ4QIKEQ4NWYZI5ZIIVLMK7GSJYOQTVK2R72EQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:43.352548983 (380h58m43.223632018s) [noise=true , address=145.79.227.15:4875 , piece_id=L7VBEME77KXSL7VAUPC4VNWAQABMUU4FKRGPTBEY5UVCHNCBNMJA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:44.390846803 (380h58m44.261929838s) [noise=true , address=145.79.227.28:4726 , piece_id=K5JT244HN4R7GZOBP3KODN4BLKFBZOZGO3MDSBU42NQ3RCX3HTFA, seg_pos={0 0}, upload=334349]: Done uploading piece. err: <nil> Dec 21 12:27:44.390866231 (380h58m44.261949276s) [seg_pos={0 0}, upload=334349]: Segment reached optimal threshold of 13 pieces. Dec 21 12:27:44.391096118 (380h58m44.262179153s) [noise=true , address=145.79.227.34:4905 , piece_id=MO6ZE4SFTFSWHK37MTLXLIOB25NZ7WSS6C7IX366OLYJ5ANG5JIQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: read tcp 188.166.225.170:33544->145.79.227.34:4905: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:27:44.391102119 (380h58m44.262185154s) [noise=true , address=145.79.227.34:4905 , piece_id=MO6ZE4SFTFSWHK37MTLXLIOB25NZ7WSS6C7IX366OLYJ5ANG5JIQ, seg_pos={0 0}, upload=334349]: Long tail cancellation due to optimal threshold Dec 21 12:27:44.391391150 (380h58m44.262474195s) [noise=true , address=145.79.227.20:4439 , piece_id=G6UALNJ7ISVIBB7ZIY63RWQ7DOL54QB6WNOQLM6E77WXWGKACQGQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: write tcp 188.166.225.170:60238->145.79.227.20:4439: use of closed network connection; write tcp 188.166.225.170:60238->145.79.227.20:4439: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60238->145.79.227.20:4439: use of closed network connection Dec 21 12:27:44.391398514 (380h58m44.262481549s) [noise=true , address=145.79.227.20:4439 , piece_id=G6UALNJ7ISVIBB7ZIY63RWQ7DOL54QB6WNOQLM6E77WXWGKACQGQ, seg_pos={0 0}, upload=334349]: Long tail cancellation due to optimal threshold Dec 21 12:27:44.391543640 (380h58m44.262626675s) [noise=true , address=145.79.227.39:4013 , piece_id=EOE6ICL7RFQNLIH4D223FGXOADJDD6RNAL6WK2YKL6ORC2QNKMFQ, seg_pos={0 0}, upload=334349]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:34860->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 12:27:44.391549221 (380h58m44.262632256s) [noise=true , address=145.79.227.39:4013 , piece_id=EOE6ICL7RFQNLIH4D223FGXOADJDD6RNAL6WK2YKL6ORC2QNKMFQ, seg_pos={0 0}, upload=334349]: Long tail cancellation due to optimal threshold Dec 21 12:27:44.391561946 (380h58m44.262644981s) [seg_pos={0 0}, upload=334349]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:27:44.391623422 (380h58m44.262706457s) [upload=334349]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:27:44.391628052 (380h58m44.262711097s) [upload=334349]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:27:44.708357059 (380h58m44.579440104s) [upload=334349]: Done uploading segments... Dec 21 12:27:44.708380202 (380h58m44.579463247s) [upload=334349]: Upload finished. err: <nil> Duration: 30.00459007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........................XX.X..X.X..X.XXX.X.X..X.........X.] duration: 5.738495827s trace: https://jaeger.dev.storj.io/trace/1ead19eea35f0048 FAIL (2025-12-21 12:26:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: SAICWYQFZB2MWKISYMZONUTTH3MDGKNFZHS63B76NJLT7JUXAFEA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 3WBZC54EBZDRIURAN4SAXXXGBIABO2CB2FQGGBRJ27HU3AMK265Q) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: SJDOQE5ZVL32BRBDOODODMFQ5A76WUZCXVZM6MN7A46PSOBMQZ5A) context deadline exceeded ======LOGS====== 21 12:26:15.924259837 (380h57m15.795342872s) [upload=334327]: Starting upload Dec 21 12:26:15.924331764 (380h57m15.795414799s) [upload=334327]: Done starting upload Dec 21 12:26:15.924420803 (380h57m15.795503838s) [upload=334327]: Uploading segments... Dec 21 12:26:15.924592039 (380h57m15.795675074s) [upload=334327]: Got next segment. Inline: false Dec 21 12:26:15.924637545 (380h57m15.79572059s) [upload=334327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:26:15.924643578 (380h57m15.795726623s) [upload=334327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:26:16.170234237 (380h57m16.041317282s) [seg_pos={0 0}, upload=334327]: Begin upload segment... Dec 21 12:26:16.170306534 (380h57m16.041389569s) [seg_pos={0 0}, upload=334327]: Done begin upload segment. Dec 21 12:26:16.170376137 (380h57m16.041459172s) [noise=true , address=145.79.227.46:4603 , piece_id=LYY27TMWP7YJWSKZQCAB3YWOE7ZJWP6H2KQ6SDMTVZMKWNVNOTGQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.170384893 (380h57m16.041467938s) [noise=true , address=145.79.227.33:4871 , piece_id=SAICWYQFZB2MWKISYMZONUTTH3MDGKNFZHS63B76NJLT7JUXAFEA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.170981911 (380h57m16.042064956s) [noise=true , address=145.79.227.27:4691 , piece_id=7KDSHKASWPEB76YFYQ74LOJI3YFNTKHKC3XYPPSWPKS23TGTSXKA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.171390659 (380h57m16.042473694s) [noise=true , address=145.79.227.44:4193 , piece_id=JF5WC243WNC5QLUSU3VGIIDQ7EEHVSAR7QACWDTHPBUJXYGLNV7A, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.172333674 (380h57m16.043416709s) [noise=true , address=145.79.227.15:4875 , piece_id=DKK3EP2LMKM5Z6EW65LBN4HB6LMJY775CJ3YQBXQORA5DLHIKHWQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.172371748 (380h57m16.043454783s) [noise=true , address=145.79.227.50:4375 , piece_id=VS5EPACX3WOQ4JQNGUPGAFPSSY4YQI5WA3LRQ34OJQA3GD4Y5XUA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.173257103 (380h57m16.044340148s) [noise=true , address=145.79.227.8:4330 , piece_id=FYESI5CIDSWNUBG4OXCPF5FD5CHX45NDDPY7EX3L2HWZTGF2VUKQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.175285876 (380h57m16.046368911s) [noise=true , address=145.79.227.42:4123 , piece_id=CHJSWRVT636OUIP2CSEYVLHCUVC6HMIS7H35TW3D4DKOEYVWXJLQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.176740346 (380h57m16.047823391s) [noise=true , address=45.91.116.12:4226 , piece_id=UT4XUMEISDZHU24JKINTLWBQ7A4V3MAHOPPZAZESSNMCEEEPDTFA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.178000595 (380h57m16.04908364s) [noise=true , address=145.79.227.51:4411 , piece_id=2DDJ5YF7IMUJR5KNCNBBIJRDEF2O2FGRKTRDKVNX6W4JQIKZGFBQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.179482237 (380h57m16.050565282s) [noise=true , address=145.79.227.36:4519 , piece_id=IEGC35PFOBW6I3GWTOXWX4245Y4QKCEZA27HMCIAKZ6MOVT2HP4A, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.181390150 (380h57m16.052473195s) [noise=true , address=145.79.227.20:4437 , piece_id=3WBZC54EBZDRIURAN4SAXXXGBIABO2CB2FQGGBRJ27HU3AMK265Q, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.183360461 (380h57m16.054443506s) [noise=true , address=145.79.227.48:4303 , piece_id=V4BPR6SKSTADNH3KWZINJ6NS2NMLQUX5JLDRAEX2WNWAB5BWKHNQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.184969345 (380h57m16.05605239s) [noise=true , address=77.90.31.10:4518 , piece_id=UEJJWEJVLEJ4XZWVLC2A67QRUKHUI2O55ET4FGLAA4VFMEKGDS6Q, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.186405931 (380h57m16.057488986s) [noise=true , address=145.79.227.29:4941 , piece_id=SKYXPA3CDGMPP42TWYZMJKWHFZ3OBQRCTWUEMDJRWF5WT2GEKZ4Q, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.186446298 (380h57m16.057529333s) [noise=true , address=45.91.116.10:4081 , piece_id=SJDOQE5ZVL32BRBDOODODMFQ5A76WUZCXVZM6MN7A46PSOBMQZ5A, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:18.050110369 (380h57m17.921193404s) [upload=334327]: Next returned nil segment Dec 21 12:26:18.050127451 (380h57m17.921210486s) [upload=334327]: Waiting for error group managing segments... Dec 21 12:26:18.374463795 (380h57m18.24554683s) [noise=true , address=145.79.227.50:4375 , piece_id=VS5EPACX3WOQ4JQNGUPGAFPSSY4YQI5WA3LRQ34OJQA3GD4Y5XUA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.401667074 (380h57m18.272750119s) [noise=true , address=145.79.227.15:4875 , piece_id=DKK3EP2LMKM5Z6EW65LBN4HB6LMJY775CJ3YQBXQORA5DLHIKHWQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.419898329 (380h57m18.290981364s) [noise=true , address=145.79.227.36:4519 , piece_id=IEGC35PFOBW6I3GWTOXWX4245Y4QKCEZA27HMCIAKZ6MOVT2HP4A, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.442843521 (380h57m18.313926556s) [noise=true , address=145.79.227.20:4437 , piece_id=3WBZC54EBZDRIURAN4SAXXXGBIABO2CB2FQGGBRJ27HU3AMK265Q, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.491649534 (380h57m18.362732569s) [noise=true , address=145.79.227.46:4603 , piece_id=LYY27TMWP7YJWSKZQCAB3YWOE7ZJWP6H2KQ6SDMTVZMKWNVNOTGQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.515905110 (380h57m18.386988145s) [noise=true , address=45.91.116.10:4081 , piece_id=SJDOQE5ZVL32BRBDOODODMFQ5A76WUZCXVZM6MN7A46PSOBMQZ5A, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.156567966 (380h57m19.027651001s) [noise=true , address=145.79.227.33:4871 , piece_id=SAICWYQFZB2MWKISYMZONUTTH3MDGKNFZHS63B76NJLT7JUXAFEA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.272490620 (380h57m19.143573655s) [noise=true , address=45.91.116.12:4226 , piece_id=UT4XUMEISDZHU24JKINTLWBQ7A4V3MAHOPPZAZESSNMCEEEPDTFA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.406879796 (380h57m19.277962841s) [noise=true , address=145.79.227.51:4411 , piece_id=2DDJ5YF7IMUJR5KNCNBBIJRDEF2O2FGRKTRDKVNX6W4JQIKZGFBQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.414777696 (380h57m19.285860731s) [noise=true , address=145.79.227.42:4123 , piece_id=CHJSWRVT636OUIP2CSEYVLHCUVC6HMIS7H35TW3D4DKOEYVWXJLQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.727298090 (380h57m19.598381125s) [noise=true , address=145.79.227.27:4691 , piece_id=7KDSHKASWPEB76YFYQ74LOJI3YFNTKHKC3XYPPSWPKS23TGTSXKA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.756636224 (380h57m19.627719259s) [noise=true , address=145.79.227.8:4330 , piece_id=FYESI5CIDSWNUBG4OXCPF5FD5CHX45NDDPY7EX3L2HWZTGF2VUKQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.801693309 (380h57m19.672776354s) [noise=true , address=145.79.227.48:4303 , piece_id=V4BPR6SKSTADNH3KWZINJ6NS2NMLQUX5JLDRAEX2WNWAB5BWKHNQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.801719008 (380h57m19.672802043s) [seg_pos={0 0}, upload=334327]: Segment reached optimal threshold of 13 pieces. Dec 21 12:26:19.802157293 (380h57m19.673240338s) [noise=true , address=145.79.227.44:4193 , piece_id=JF5WC243WNC5QLUSU3VGIIDQ7EEHVSAR7QACWDTHPBUJXYGLNV7A, seg_pos={0 0}, upload=334327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 21 12:26:19.802168294 (380h57m19.673251339s) [noise=true , address=145.79.227.44:4193 , piece_id=JF5WC243WNC5QLUSU3VGIIDQ7EEHVSAR7QACWDTHPBUJXYGLNV7A, seg_pos={0 0}, upload=334327]: Long tail cancellation due to optimal threshold Dec 21 12:26:19.802448748 (380h57m19.673531803s) [noise=true , address=77.90.31.10:4518 , piece_id=UEJJWEJVLEJ4XZWVLC2A67QRUKHUI2O55ET4FGLAA4VFMEKGDS6Q, seg_pos={0 0}, upload=334327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:54716->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:54716->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54716->77.90.31.10:4518: use of closed network connection Dec 21 12:26:19.802464497 (380h57m19.673547542s) [noise=true , address=77.90.31.10:4518 , piece_id=UEJJWEJVLEJ4XZWVLC2A67QRUKHUI2O55ET4FGLAA4VFMEKGDS6Q, seg_pos={0 0}, upload=334327]: Long tail cancellation due to optimal threshold Dec 21 12:26:19.802590117 (380h57m19.673673162s) [noise=true , address=145.79.227.29:4941 , piece_id=SKYXPA3CDGMPP42TWYZMJKWHFZ3OBQRCTWUEMDJRWF5WT2GEKZ4Q, seg_pos={0 0}, upload=334327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 21 12:26:19.802600817 (380h57m19.673683872s) [noise=true , address=145.79.227.29:4941 , piece_id=SKYXPA3CDGMPP42TWYZMJKWHFZ3OBQRCTWUEMDJRWF5WT2GEKZ4Q, seg_pos={0 0}, upload=334327]: Long tail cancellation due to optimal threshold Dec 21 12:26:19.802624392 (380h57m19.673707427s) [seg_pos={0 0}, upload=334327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:26:19.802773586 (380h57m19.673856631s) [upload=334327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:26:19.802781541 (380h57m19.673864586s) [upload=334327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:26:20.096121294 (380h57m19.967204329s) [upload=334327]: Done uploading segments... Dec 21 12:26:20.096140610 (380h57m19.967223645s) [upload=334327]: Upload finished. err: <nil> Duration: 5.738512849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........................XX.X..X.X..X.XXX.X.X..X.........X] duration: 30.001840992s trace: https://jaeger.dev.storj.io/trace/1ead19eea35f0048 FAIL (2025-12-21 12:26:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx, Piece ID: SAICWYQFZB2MWKISYMZONUTTH3MDGKNFZHS63B76NJLT7JUXAFEA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: 3WBZC54EBZDRIURAN4SAXXXGBIABO2CB2FQGGBRJ27HU3AMK265Q) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: SJDOQE5ZVL32BRBDOODODMFQ5A76WUZCXVZM6MN7A46PSOBMQZ5A) context deadline exceeded ======LOGS====== 21 12:26:15.924259837 (380h57m15.795342872s) [upload=334327]: Starting upload Dec 21 12:26:15.924331764 (380h57m15.795414799s) [upload=334327]: Done starting upload Dec 21 12:26:15.924420803 (380h57m15.795503838s) [upload=334327]: Uploading segments... Dec 21 12:26:15.924592039 (380h57m15.795675074s) [upload=334327]: Got next segment. Inline: false Dec 21 12:26:15.924637545 (380h57m15.79572059s) [upload=334327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:26:15.924643578 (380h57m15.795726623s) [upload=334327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:26:16.170234237 (380h57m16.041317282s) [seg_pos={0 0}, upload=334327]: Begin upload segment... Dec 21 12:26:16.170306534 (380h57m16.041389569s) [seg_pos={0 0}, upload=334327]: Done begin upload segment. Dec 21 12:26:16.170376137 (380h57m16.041459172s) [noise=true , address=145.79.227.46:4603 , piece_id=LYY27TMWP7YJWSKZQCAB3YWOE7ZJWP6H2KQ6SDMTVZMKWNVNOTGQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.170384893 (380h57m16.041467938s) [noise=true , address=145.79.227.33:4871 , piece_id=SAICWYQFZB2MWKISYMZONUTTH3MDGKNFZHS63B76NJLT7JUXAFEA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.170981911 (380h57m16.042064956s) [noise=true , address=145.79.227.27:4691 , piece_id=7KDSHKASWPEB76YFYQ74LOJI3YFNTKHKC3XYPPSWPKS23TGTSXKA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.171390659 (380h57m16.042473694s) [noise=true , address=145.79.227.44:4193 , piece_id=JF5WC243WNC5QLUSU3VGIIDQ7EEHVSAR7QACWDTHPBUJXYGLNV7A, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.172333674 (380h57m16.043416709s) [noise=true , address=145.79.227.15:4875 , piece_id=DKK3EP2LMKM5Z6EW65LBN4HB6LMJY775CJ3YQBXQORA5DLHIKHWQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.172371748 (380h57m16.043454783s) [noise=true , address=145.79.227.50:4375 , piece_id=VS5EPACX3WOQ4JQNGUPGAFPSSY4YQI5WA3LRQ34OJQA3GD4Y5XUA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.173257103 (380h57m16.044340148s) [noise=true , address=145.79.227.8:4330 , piece_id=FYESI5CIDSWNUBG4OXCPF5FD5CHX45NDDPY7EX3L2HWZTGF2VUKQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.175285876 (380h57m16.046368911s) [noise=true , address=145.79.227.42:4123 , piece_id=CHJSWRVT636OUIP2CSEYVLHCUVC6HMIS7H35TW3D4DKOEYVWXJLQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.176740346 (380h57m16.047823391s) [noise=true , address=45.91.116.12:4226 , piece_id=UT4XUMEISDZHU24JKINTLWBQ7A4V3MAHOPPZAZESSNMCEEEPDTFA, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.178000595 (380h57m16.04908364s) [noise=true , address=145.79.227.51:4411 , piece_id=2DDJ5YF7IMUJR5KNCNBBIJRDEF2O2FGRKTRDKVNX6W4JQIKZGFBQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.179482237 (380h57m16.050565282s) [noise=true , address=145.79.227.36:4519 , piece_id=IEGC35PFOBW6I3GWTOXWX4245Y4QKCEZA27HMCIAKZ6MOVT2HP4A, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.181390150 (380h57m16.052473195s) [noise=true , address=145.79.227.20:4437 , piece_id=3WBZC54EBZDRIURAN4SAXXXGBIABO2CB2FQGGBRJ27HU3AMK265Q, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.183360461 (380h57m16.054443506s) [noise=true , address=145.79.227.48:4303 , piece_id=V4BPR6SKSTADNH3KWZINJ6NS2NMLQUX5JLDRAEX2WNWAB5BWKHNQ, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.184969345 (380h57m16.05605239s) [noise=true , address=77.90.31.10:4518 , piece_id=UEJJWEJVLEJ4XZWVLC2A67QRUKHUI2O55ET4FGLAA4VFMEKGDS6Q, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.186405931 (380h57m16.057488986s) [noise=true , address=145.79.227.29:4941 , piece_id=SKYXPA3CDGMPP42TWYZMJKWHFZ3OBQRCTWUEMDJRWF5WT2GEKZ4Q, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:16.186446298 (380h57m16.057529333s) [noise=true , address=45.91.116.10:4081 , piece_id=SJDOQE5ZVL32BRBDOODODMFQ5A76WUZCXVZM6MN7A46PSOBMQZ5A, seg_pos={0 0}, upload=334327]: Uploading piece... Dec 21 12:26:18.050110369 (380h57m17.921193404s) [upload=334327]: Next returned nil segment Dec 21 12:26:18.050127451 (380h57m17.921210486s) [upload=334327]: Waiting for error group managing segments... Dec 21 12:26:18.374463795 (380h57m18.24554683s) [noise=true , address=145.79.227.50:4375 , piece_id=VS5EPACX3WOQ4JQNGUPGAFPSSY4YQI5WA3LRQ34OJQA3GD4Y5XUA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.401667074 (380h57m18.272750119s) [noise=true , address=145.79.227.15:4875 , piece_id=DKK3EP2LMKM5Z6EW65LBN4HB6LMJY775CJ3YQBXQORA5DLHIKHWQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.419898329 (380h57m18.290981364s) [noise=true , address=145.79.227.36:4519 , piece_id=IEGC35PFOBW6I3GWTOXWX4245Y4QKCEZA27HMCIAKZ6MOVT2HP4A, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.442843521 (380h57m18.313926556s) [noise=true , address=145.79.227.20:4437 , piece_id=3WBZC54EBZDRIURAN4SAXXXGBIABO2CB2FQGGBRJ27HU3AMK265Q, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.491649534 (380h57m18.362732569s) [noise=true , address=145.79.227.46:4603 , piece_id=LYY27TMWP7YJWSKZQCAB3YWOE7ZJWP6H2KQ6SDMTVZMKWNVNOTGQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:18.515905110 (380h57m18.386988145s) [noise=true , address=45.91.116.10:4081 , piece_id=SJDOQE5ZVL32BRBDOODODMFQ5A76WUZCXVZM6MN7A46PSOBMQZ5A, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.156567966 (380h57m19.027651001s) [noise=true , address=145.79.227.33:4871 , piece_id=SAICWYQFZB2MWKISYMZONUTTH3MDGKNFZHS63B76NJLT7JUXAFEA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.272490620 (380h57m19.143573655s) [noise=true , address=45.91.116.12:4226 , piece_id=UT4XUMEISDZHU24JKINTLWBQ7A4V3MAHOPPZAZESSNMCEEEPDTFA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.406879796 (380h57m19.277962841s) [noise=true , address=145.79.227.51:4411 , piece_id=2DDJ5YF7IMUJR5KNCNBBIJRDEF2O2FGRKTRDKVNX6W4JQIKZGFBQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.414777696 (380h57m19.285860731s) [noise=true , address=145.79.227.42:4123 , piece_id=CHJSWRVT636OUIP2CSEYVLHCUVC6HMIS7H35TW3D4DKOEYVWXJLQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.727298090 (380h57m19.598381125s) [noise=true , address=145.79.227.27:4691 , piece_id=7KDSHKASWPEB76YFYQ74LOJI3YFNTKHKC3XYPPSWPKS23TGTSXKA, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.756636224 (380h57m19.627719259s) [noise=true , address=145.79.227.8:4330 , piece_id=FYESI5CIDSWNUBG4OXCPF5FD5CHX45NDDPY7EX3L2HWZTGF2VUKQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.801693309 (380h57m19.672776354s) [noise=true , address=145.79.227.48:4303 , piece_id=V4BPR6SKSTADNH3KWZINJ6NS2NMLQUX5JLDRAEX2WNWAB5BWKHNQ, seg_pos={0 0}, upload=334327]: Done uploading piece. err: <nil> Dec 21 12:26:19.801719008 (380h57m19.672802043s) [seg_pos={0 0}, upload=334327]: Segment reached optimal threshold of 13 pieces. Dec 21 12:26:19.802157293 (380h57m19.673240338s) [noise=true , address=145.79.227.44:4193 , piece_id=JF5WC243WNC5QLUSU3VGIIDQ7EEHVSAR7QACWDTHPBUJXYGLNV7A, seg_pos={0 0}, upload=334327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 21 12:26:19.802168294 (380h57m19.673251339s) [noise=true , address=145.79.227.44:4193 , piece_id=JF5WC243WNC5QLUSU3VGIIDQ7EEHVSAR7QACWDTHPBUJXYGLNV7A, seg_pos={0 0}, upload=334327]: Long tail cancellation due to optimal threshold Dec 21 12:26:19.802448748 (380h57m19.673531803s) [noise=true , address=77.90.31.10:4518 , piece_id=UEJJWEJVLEJ4XZWVLC2A67QRUKHUI2O55ET4FGLAA4VFMEKGDS6Q, seg_pos={0 0}, upload=334327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:54716->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:54716->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54716->77.90.31.10:4518: use of closed network connection Dec 21 12:26:19.802464497 (380h57m19.673547542s) [noise=true , address=77.90.31.10:4518 , piece_id=UEJJWEJVLEJ4XZWVLC2A67QRUKHUI2O55ET4FGLAA4VFMEKGDS6Q, seg_pos={0 0}, upload=334327]: Long tail cancellation due to optimal threshold Dec 21 12:26:19.802590117 (380h57m19.673673162s) [noise=true , address=145.79.227.29:4941 , piece_id=SKYXPA3CDGMPP42TWYZMJKWHFZ3OBQRCTWUEMDJRWF5WT2GEKZ4Q, seg_pos={0 0}, upload=334327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 21 12:26:19.802600817 (380h57m19.673683872s) [noise=true , address=145.79.227.29:4941 , piece_id=SKYXPA3CDGMPP42TWYZMJKWHFZ3OBQRCTWUEMDJRWF5WT2GEKZ4Q, seg_pos={0 0}, upload=334327]: Long tail cancellation due to optimal threshold Dec 21 12:26:19.802624392 (380h57m19.673707427s) [seg_pos={0 0}, upload=334327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:26:19.802773586 (380h57m19.673856631s) [upload=334327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:26:19.802781541 (380h57m19.673864586s) [upload=334327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:26:20.096121294 (380h57m19.967204329s) [upload=334327]: Done uploading segments... Dec 21 12:26:20.096140610 (380h57m19.967223645s) [upload=334327]: Upload finished. err: <nil> Duration: 30.001864348s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............................XX.X..X.X..X.XXX.X.X..X.........] duration: 5.963084747s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:25:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.963110407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............................XX.X..X.X..X.XXX.X.X..X........] duration: 5.948105897s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:24:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.948135432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............................XX.X..X.X..X.XXX.X.X..X.......] duration: 5.729896306s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.729921833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............................XX.X..X.X..X.XXX.X.X..X......] duration: 5.736645032s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:23:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.736743179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................XX.X..X.X..X.XXX.X.X..X.....] duration: 5.968498631s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:22:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.968526094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................XX.X..X.X..X.XXX.X.X..X....] duration: 5.688158046s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:21:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.68818157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..................................XX.X..X.X..X.XXX.X.X..X...] duration: 5.906379412s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:20:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.906410311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...................................XX.X..X.X..X.XXX.X.X..X..] duration: 5.800992437s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:19:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.801021592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....................................XX.X..X.X..X.XXX.X.X..X.] duration: 5.753516019s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 5.753550494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....................................XX.X..X.X..X.XXX.X.X..X] duration: 30.003326108s trace: https://jaeger.dev.storj.io/trace/53a0498999348072 FAIL (2025-12-21 12:19:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: 3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q) context deadline exceeded ======LOGS====== .. Dec 21 12:18:38.843985236 (380h49m38.715068271s) [upload=334215]: Got next segment. Inline: false Dec 21 12:18:38.844026095 (380h49m38.71510914s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:18:38.844031395 (380h49m38.71511443s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:18:39.104374757 (380h49m38.975457802s) [seg_pos={0 0}, upload=334215]: Begin upload segment... Dec 21 12:18:39.104426174 (380h49m38.975509219s) [seg_pos={0 0}, upload=334215]: Done begin upload segment. Dec 21 12:18:39.104457043 (380h49m38.975540078s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.104544440 (380h49m38.975627475s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105210137 (380h49m38.976293172s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.105910421 (380h49m38.976993456s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106117615 (380h49m38.97720065s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106291637 (380h49m38.977374682s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106698801 (380h49m38.977781846s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.106727186 (380h49m38.977810231s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107067534 (380h49m38.978150579s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107220535 (380h49m38.97830357s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107454611 (380h49m38.978537646s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107709426 (380h49m38.978792461s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.107825457 (380h49m38.978908492s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108206463 (380h49m38.979289498s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.108661790 (380h49m38.979744825s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:39.109120142 (380h49m38.980203188s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Uploading piece... Dec 21 12:18:40.964149150 (380h49m40.835232215s) [upload=334215]: Next returned nil segment Dec 21 12:18:40.964164429 (380h49m40.835247474s) [upload=334215]: Waiting for error group managing segments... Dec 21 12:18:41.276070043 (380h49m41.147153078s) [noise=true , address=145.79.227.20:4439 , piece_id=ZRIA22MR2IQHBMOTMCFOXHBBSGFUPCVZDACNTDSQREI7EMFV2UFQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.284419041 (380h49m41.155502086s) [noise=true , address=145.79.227.8:4330 , piece_id=2SNNHRRJEXCDZNVWYDKPW4O5AHGDX5SAH6BK4KVIE5UFS3EUTSZQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.286723389 (380h49m41.157806424s) [noise=true , address=145.79.227.41:4086 , piece_id=N7VDDVBXPMFJVIRJHCXJELB4OADGTPDTRZ254KNDJUCAOIJ6ADNQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.295439235 (380h49m41.16652228s) [noise=true , address=145.79.227.32:4834 , piece_id=KYBW4WCJWSPALLOUG4564EHS3UJXX55E32LQV75EAQBWEFZ6JQLQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.305127202 (380h49m41.176210237s) [noise=true , address=145.79.227.36:4519 , piece_id=ADVF52ZEV3YDJCZRWWWNSSQAHWD25NNTX33A3KGHR42VVVGXXQMQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.310170485 (380h49m41.18125352s) [noise=true , address=145.79.227.43:4157 , piece_id=HNZLL244KMGJRP5E4L2ANB27BZITCFWB7PM4Y2EXL6TLVNDIHUXQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.313019389 (380h49m41.184102424s) [noise=true , address=77.90.31.14:4663 , piece_id=RDPZQXDWR43FZE6G4XHPRDJ3LEO7MEG4Z5GP77JXNQJGCGKECEJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.328477083 (380h49m41.199560118s) [noise=true , address=145.79.227.46:4603 , piece_id=3FIJKDSMRYFBQI6IWVD5UG2K7PFWRGVNYHYNLACWWRDGXCMIKY3Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.329399520 (380h49m41.200482565s) [noise=true , address=145.79.227.29:4941 , piece_id=U6N2RRWFSNRCRN3KPB5VP5R7AZ23D4PCTTBLW5KDAVQDLGYDU6CA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.347500306 (380h49m41.218583341s) [noise=true , address=145.79.227.35:4481 , piece_id=AL3JTSYWW5RAHSEMAIUC6YXM4FZP6MA4LGA27GN6HFS6I7ETRT5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.372911513 (380h49m41.243994558s) [noise=true , address=145.79.227.27:4690 , piece_id=MOXVGVAPRTR37PSOIWAYVOFQDWI25YZQTHQSAB3HVBAX72UXJUSA, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.591644483 (380h49m41.462727518s) [noise=true , address=145.79.227.37:4565 , piece_id=O2V7KTJROA6KO7HLE37PG74O2NEB7QZ4UBKVFX7FDK55R3QTOA5Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646441775 (380h49m41.51752482s) [noise=true , address=45.91.116.12:4226 , piece_id=E7GQUHAABWEBNAW6H3Q3CKZYZWDM4SKQEQ4R7EFQC3QLKO5IGZBQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: <nil> Dec 21 12:18:41.646458186 (380h49m41.517541231s) [seg_pos={0 0}, upload=334215]: Segment reached optimal threshold of 13 pieces. Dec 21 12:18:41.646627178 (380h49m41.517710213s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53508->145.79.227.38:4977: use of closed network connection Dec 21 12:18:41.646637258 (380h49m41.517720303s) [noise=true , address=145.79.227.38:4977 , piece_id=XWIDIXOM65AJL33KQR4HL64IFVARJSFDLJOHQUCHMXBO53WOUK2Q, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646648699 (380h49m41.517731734s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:18:41.646652817 (380h49m41.517735862s) [noise=true , address=145.79.227.47:4265 , piece_id=RK2A4PJC247JLNSLMMSSLJZVTI6JAWD2I4ITZJIMCRUZ2ULTJY3A, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646745814 (380h49m41.517828849s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo): protocol: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39614->145.79.227.52:4445: use of closed network connection Dec 21 12:18:41.646826297 (380h49m41.517909362s) [noise=true , address=145.79.227.52:4445 , piece_id=GLWHZ6CURFNADNFS3WLWPVHBCETKYXJO3GS57GZLAUXORV64CDJQ, seg_pos={0 0}, upload=334215]: Long tail cancellation due to optimal threshold Dec 21 12:18:41.646849562 (380h49m41.517932607s) [seg_pos={0 0}, upload=334215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:18:41.646974048 (380h49m41.518057083s) [upload=334215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:18:41.646979889 (380h49m41.518062924s) [upload=334215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:18:41.960328471 (380h49m41.831411537s) [upload=334215]: Done uploading segments... Dec 21 12:18:41.960398725 (380h49m41.83148176s) [upload=334215]: Upload finished. err: <nil> Duration: 30.00335274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......................................XX.X..X.X..X.XXX.X.X..] duration: 9.052582481s trace: https://jaeger.dev.storj.io/trace/801836218333336 FAIL (2025-12-21 12:18:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:16:52.430679889 (380h47m52.301762934s) [upload=334188]: Starting upload Dec 21 12:16:52.430798605 (380h47m52.30188165s) [upload=334188]: Done starting upload Dec 21 12:16:52.431011420 (380h47m52.302094455s) [upload=334188]: Uploading segments... Dec 21 12:16:52.431070162 (380h47m52.302153197s) [upload=334188]: Got next segment. Inline: false Dec 21 12:16:52.431101041 (380h47m52.302184076s) [upload=334188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:16:52.431105580 (380h47m52.302188625s) [upload=334188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:16:54.386198434 (380h47m54.25728149s) [seg_pos={0 0}, upload=334188]: Begin upload segment... Dec 21 12:16:54.386255854 (380h47m54.257338889s) [seg_pos={0 0}, upload=334188]: Done begin upload segment. Dec 21 12:16:54.386284649 (380h47m54.257367694s) [noise=true , address=145.79.227.32:4835 , piece_id=DHQE5HL65NNZSIDCCIKASKM2BZUCCS5NXGW4VF7XWAARNCFUWOBQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.387626253 (380h47m54.258709288s) [noise=true , address=45.91.116.13:4153 , piece_id=GO5RMG74Z7VXFESGC6YGYYNOWPUUHQESTSFTCLCGJYFMRO7J6FMQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.388404386 (380h47m54.259487431s) [noise=true , address=145.79.227.48:4301 , piece_id=H2QPVXSJUSISVILZ56KA7B3K6WNPTO35TGCJMQUBKCGAEDLBRLHA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.388825608 (380h47m54.259908653s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.389159342 (380h47m54.260242378s) [noise=true , address=45.91.116.8:4047 , piece_id=UKQSJFOIWNPJPIRULJG2ABLRH2KGW4H6LXKBLTE63UEBEADJ2YYQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.389623908 (380h47m54.260706953s) [noise=true , address=145.79.227.39:4014 , piece_id=DIM6VYTUDPHYSI4OY7VEEZJB45SZWETSL6DIRIUTN4COPNEB3BNQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.390915798 (380h47m54.261998833s) [noise=true , address=145.79.227.46:4602 , piece_id=3F44IPNW6ADFR532FOCSF32RYCJF6UTI3BAYPHYZ32EAUN6IXT5A, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.391063709 (380h47m54.262146754s) [noise=true , address=145.79.227.43:4157 , piece_id=DW2XREE5F6IEPEYELDB3ZYARTXQOD6FH3DAH2ACMESP2RH4L3POA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.392035429 (380h47m54.263118464s) [noise=true , address=45.91.116.7:4009 , piece_id=6FEGEJDSB4NXVN5UDWYPENXYJZNBFP26KFAT7KIZEWUBDKEIIXUQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.392311255 (380h47m54.2633943s) [noise=true , address=145.79.227.42:4123 , piece_id=5DBX2SBAZV6NUPC7J7MSNW5OTXLQRWFLKYA6Y6IOHSD7K46QZBDA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.393169468 (380h47m54.264252513s) [noise=true , address=145.79.227.37:4566 , piece_id=BJD2RWGEMBFV3LANG5SKQP2M45UF3TH6LTJ5Q5JZ5H5RFRR6V55A, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.393483707 (380h47m54.264566742s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.394476878 (380h47m54.265559913s) [noise=true , address=77.90.31.17:4770 , piece_id=LSAQNVQV6ZEOKKBODN5FTVCGKPDN3D344NZ5HYNQA6TLUJBFDT3Q, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.394873383 (380h47m54.265956418s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.403422182 (380h47m54.274505217s) [noise=true , address=145.79.227.49:4339 , piece_id=6UIYCLJ2DXDKDLB6Q5TWUYENTTEASGE2VGCHF4HLK4JN2NQJO7HA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.403874753 (380h47m54.274957788s) [noise=true , address=145.79.227.50:4374 , piece_id=YCR7XDSIAXRTUAKDMXR7PTJ73AJ7KPRBNICCQZLSSUV7AP5OVAHQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:56.246383992 (380h47m56.117467027s) [upload=334188]: Next returned nil segment Dec 21 12:16:56.246409850 (380h47m56.117492885s) [upload=334188]: Waiting for error group managing segments... Dec 21 12:16:56.576072878 (380h47m56.447155933s) [noise=true , address=145.79.227.32:4835 , piece_id=DHQE5HL65NNZSIDCCIKASKM2BZUCCS5NXGW4VF7XWAARNCFUWOBQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.598667150 (380h47m56.469750185s) [noise=true , address=145.79.227.37:4566 , piece_id=BJD2RWGEMBFV3LANG5SKQP2M45UF3TH6LTJ5Q5JZ5H5RFRR6V55A, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.646458722 (380h47m56.517541767s) [noise=true , address=145.79.227.43:4157 , piece_id=DW2XREE5F6IEPEYELDB3ZYARTXQOD6FH3DAH2ACMESP2RH4L3POA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.679261544 (380h47m56.550344579s) [noise=true , address=77.90.31.17:4770 , piece_id=LSAQNVQV6ZEOKKBODN5FTVCGKPDN3D344NZ5HYNQA6TLUJBFDT3Q, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.688404533 (380h47m56.559487579s) [noise=true , address=45.91.116.8:4047 , piece_id=UKQSJFOIWNPJPIRULJG2ABLRH2KGW4H6LXKBLTE63UEBEADJ2YYQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.697132984 (380h47m56.568216019s) [noise=true , address=45.91.116.13:4153 , piece_id=GO5RMG74Z7VXFESGC6YGYYNOWPUUHQESTSFTCLCGJYFMRO7J6FMQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.698716369 (380h47m56.569799404s) [noise=true , address=45.91.116.7:4009 , piece_id=6FEGEJDSB4NXVN5UDWYPENXYJZNBFP26KFAT7KIZEWUBDKEIIXUQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.776653302 (380h47m56.647736337s) [noise=true , address=145.79.227.39:4014 , piece_id=DIM6VYTUDPHYSI4OY7VEEZJB45SZWETSL6DIRIUTN4COPNEB3BNQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.861714852 (380h47m56.732797887s) [noise=true , address=145.79.227.50:4374 , piece_id=YCR7XDSIAXRTUAKDMXR7PTJ73AJ7KPRBNICCQZLSSUV7AP5OVAHQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.870445466 (380h47m56.741528511s) [noise=true , address=145.79.227.46:4602 , piece_id=3F44IPNW6ADFR532FOCSF32RYCJF6UTI3BAYPHYZ32EAUN6IXT5A, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.278845830 (380h47m57.149928865s) [noise=true , address=145.79.227.49:4339 , piece_id=6UIYCLJ2DXDKDLB6Q5TWUYENTTEASGE2VGCHF4HLK4JN2NQJO7HA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.315489715 (380h47m57.18657276s) [noise=true , address=145.79.227.42:4123 , piece_id=5DBX2SBAZV6NUPC7J7MSNW5OTXLQRWFLKYA6Y6IOHSD7K46QZBDA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.388767035 (380h47m57.259850081s) [noise=true , address=145.79.227.48:4301 , piece_id=H2QPVXSJUSISVILZ56KA7B3K6WNPTO35TGCJMQUBKCGAEDLBRLHA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.388804847 (380h47m57.259887892s) [seg_pos={0 0}, upload=334188]: Segment reached optimal threshold of 13 pieces. Dec 21 12:16:57.389045496 (380h47m57.260128541s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389054282 (380h47m57.260137337s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389244826 (380h47m57.260327861s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389281535 (380h47m57.26036458s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389354815 (380h47m57.26043785s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389365345 (380h47m57.2604484s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389391785 (380h47m57.26047483s) [seg_pos={0 0}, upload=334188]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:16:57.389478741 (380h47m57.260561786s) [upload=334188]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:16:57.389485083 (380h47m57.260568128s) [upload=334188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:16:57.707610560 (380h47m57.578693595s) [upload=334188]: Done uploading segments... Dec 21 12:16:57.707629265 (380h47m57.5787123s) [upload=334188]: Upload finished. err: <nil> Duration: 9.052609652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......................................XX.X..X.X..X.XXX.X.X.] duration: 15.183347111s trace: https://jaeger.dev.storj.io/trace/801836218333336 FAIL (2025-12-21 12:17:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:16:52.430679889 (380h47m52.301762934s) [upload=334188]: Starting upload Dec 21 12:16:52.430798605 (380h47m52.30188165s) [upload=334188]: Done starting upload Dec 21 12:16:52.431011420 (380h47m52.302094455s) [upload=334188]: Uploading segments... Dec 21 12:16:52.431070162 (380h47m52.302153197s) [upload=334188]: Got next segment. Inline: false Dec 21 12:16:52.431101041 (380h47m52.302184076s) [upload=334188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:16:52.431105580 (380h47m52.302188625s) [upload=334188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:16:54.386198434 (380h47m54.25728149s) [seg_pos={0 0}, upload=334188]: Begin upload segment... Dec 21 12:16:54.386255854 (380h47m54.257338889s) [seg_pos={0 0}, upload=334188]: Done begin upload segment. Dec 21 12:16:54.386284649 (380h47m54.257367694s) [noise=true , address=145.79.227.32:4835 , piece_id=DHQE5HL65NNZSIDCCIKASKM2BZUCCS5NXGW4VF7XWAARNCFUWOBQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.387626253 (380h47m54.258709288s) [noise=true , address=45.91.116.13:4153 , piece_id=GO5RMG74Z7VXFESGC6YGYYNOWPUUHQESTSFTCLCGJYFMRO7J6FMQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.388404386 (380h47m54.259487431s) [noise=true , address=145.79.227.48:4301 , piece_id=H2QPVXSJUSISVILZ56KA7B3K6WNPTO35TGCJMQUBKCGAEDLBRLHA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.388825608 (380h47m54.259908653s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.389159342 (380h47m54.260242378s) [noise=true , address=45.91.116.8:4047 , piece_id=UKQSJFOIWNPJPIRULJG2ABLRH2KGW4H6LXKBLTE63UEBEADJ2YYQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.389623908 (380h47m54.260706953s) [noise=true , address=145.79.227.39:4014 , piece_id=DIM6VYTUDPHYSI4OY7VEEZJB45SZWETSL6DIRIUTN4COPNEB3BNQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.390915798 (380h47m54.261998833s) [noise=true , address=145.79.227.46:4602 , piece_id=3F44IPNW6ADFR532FOCSF32RYCJF6UTI3BAYPHYZ32EAUN6IXT5A, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.391063709 (380h47m54.262146754s) [noise=true , address=145.79.227.43:4157 , piece_id=DW2XREE5F6IEPEYELDB3ZYARTXQOD6FH3DAH2ACMESP2RH4L3POA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.392035429 (380h47m54.263118464s) [noise=true , address=45.91.116.7:4009 , piece_id=6FEGEJDSB4NXVN5UDWYPENXYJZNBFP26KFAT7KIZEWUBDKEIIXUQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.392311255 (380h47m54.2633943s) [noise=true , address=145.79.227.42:4123 , piece_id=5DBX2SBAZV6NUPC7J7MSNW5OTXLQRWFLKYA6Y6IOHSD7K46QZBDA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.393169468 (380h47m54.264252513s) [noise=true , address=145.79.227.37:4566 , piece_id=BJD2RWGEMBFV3LANG5SKQP2M45UF3TH6LTJ5Q5JZ5H5RFRR6V55A, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.393483707 (380h47m54.264566742s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.394476878 (380h47m54.265559913s) [noise=true , address=77.90.31.17:4770 , piece_id=LSAQNVQV6ZEOKKBODN5FTVCGKPDN3D344NZ5HYNQA6TLUJBFDT3Q, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.394873383 (380h47m54.265956418s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.403422182 (380h47m54.274505217s) [noise=true , address=145.79.227.49:4339 , piece_id=6UIYCLJ2DXDKDLB6Q5TWUYENTTEASGE2VGCHF4HLK4JN2NQJO7HA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.403874753 (380h47m54.274957788s) [noise=true , address=145.79.227.50:4374 , piece_id=YCR7XDSIAXRTUAKDMXR7PTJ73AJ7KPRBNICCQZLSSUV7AP5OVAHQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:56.246383992 (380h47m56.117467027s) [upload=334188]: Next returned nil segment Dec 21 12:16:56.246409850 (380h47m56.117492885s) [upload=334188]: Waiting for error group managing segments... Dec 21 12:16:56.576072878 (380h47m56.447155933s) [noise=true , address=145.79.227.32:4835 , piece_id=DHQE5HL65NNZSIDCCIKASKM2BZUCCS5NXGW4VF7XWAARNCFUWOBQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.598667150 (380h47m56.469750185s) [noise=true , address=145.79.227.37:4566 , piece_id=BJD2RWGEMBFV3LANG5SKQP2M45UF3TH6LTJ5Q5JZ5H5RFRR6V55A, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.646458722 (380h47m56.517541767s) [noise=true , address=145.79.227.43:4157 , piece_id=DW2XREE5F6IEPEYELDB3ZYARTXQOD6FH3DAH2ACMESP2RH4L3POA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.679261544 (380h47m56.550344579s) [noise=true , address=77.90.31.17:4770 , piece_id=LSAQNVQV6ZEOKKBODN5FTVCGKPDN3D344NZ5HYNQA6TLUJBFDT3Q, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.688404533 (380h47m56.559487579s) [noise=true , address=45.91.116.8:4047 , piece_id=UKQSJFOIWNPJPIRULJG2ABLRH2KGW4H6LXKBLTE63UEBEADJ2YYQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.697132984 (380h47m56.568216019s) [noise=true , address=45.91.116.13:4153 , piece_id=GO5RMG74Z7VXFESGC6YGYYNOWPUUHQESTSFTCLCGJYFMRO7J6FMQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.698716369 (380h47m56.569799404s) [noise=true , address=45.91.116.7:4009 , piece_id=6FEGEJDSB4NXVN5UDWYPENXYJZNBFP26KFAT7KIZEWUBDKEIIXUQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.776653302 (380h47m56.647736337s) [noise=true , address=145.79.227.39:4014 , piece_id=DIM6VYTUDPHYSI4OY7VEEZJB45SZWETSL6DIRIUTN4COPNEB3BNQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.861714852 (380h47m56.732797887s) [noise=true , address=145.79.227.50:4374 , piece_id=YCR7XDSIAXRTUAKDMXR7PTJ73AJ7KPRBNICCQZLSSUV7AP5OVAHQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.870445466 (380h47m56.741528511s) [noise=true , address=145.79.227.46:4602 , piece_id=3F44IPNW6ADFR532FOCSF32RYCJF6UTI3BAYPHYZ32EAUN6IXT5A, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.278845830 (380h47m57.149928865s) [noise=true , address=145.79.227.49:4339 , piece_id=6UIYCLJ2DXDKDLB6Q5TWUYENTTEASGE2VGCHF4HLK4JN2NQJO7HA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.315489715 (380h47m57.18657276s) [noise=true , address=145.79.227.42:4123 , piece_id=5DBX2SBAZV6NUPC7J7MSNW5OTXLQRWFLKYA6Y6IOHSD7K46QZBDA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.388767035 (380h47m57.259850081s) [noise=true , address=145.79.227.48:4301 , piece_id=H2QPVXSJUSISVILZ56KA7B3K6WNPTO35TGCJMQUBKCGAEDLBRLHA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.388804847 (380h47m57.259887892s) [seg_pos={0 0}, upload=334188]: Segment reached optimal threshold of 13 pieces. Dec 21 12:16:57.389045496 (380h47m57.260128541s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389054282 (380h47m57.260137337s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389244826 (380h47m57.260327861s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389281535 (380h47m57.26036458s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389354815 (380h47m57.26043785s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389365345 (380h47m57.2604484s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389391785 (380h47m57.26047483s) [seg_pos={0 0}, upload=334188]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:16:57.389478741 (380h47m57.260561786s) [upload=334188]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:16:57.389485083 (380h47m57.260568128s) [upload=334188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:16:57.707610560 (380h47m57.578693595s) [upload=334188]: Done uploading segments... Dec 21 12:16:57.707629265 (380h47m57.5787123s) [upload=334188]: Upload finished. err: <nil> Duration: 15.183373172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........................................XX.X..X.X..X.XXX.X.X] duration: 30.003412038s trace: https://jaeger.dev.storj.io/trace/801836218333336 FAIL (2025-12-21 12:17:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:16:52.430679889 (380h47m52.301762934s) [upload=334188]: Starting upload Dec 21 12:16:52.430798605 (380h47m52.30188165s) [upload=334188]: Done starting upload Dec 21 12:16:52.431011420 (380h47m52.302094455s) [upload=334188]: Uploading segments... Dec 21 12:16:52.431070162 (380h47m52.302153197s) [upload=334188]: Got next segment. Inline: false Dec 21 12:16:52.431101041 (380h47m52.302184076s) [upload=334188]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:16:52.431105580 (380h47m52.302188625s) [upload=334188]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:16:54.386198434 (380h47m54.25728149s) [seg_pos={0 0}, upload=334188]: Begin upload segment... Dec 21 12:16:54.386255854 (380h47m54.257338889s) [seg_pos={0 0}, upload=334188]: Done begin upload segment. Dec 21 12:16:54.386284649 (380h47m54.257367694s) [noise=true , address=145.79.227.32:4835 , piece_id=DHQE5HL65NNZSIDCCIKASKM2BZUCCS5NXGW4VF7XWAARNCFUWOBQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.387626253 (380h47m54.258709288s) [noise=true , address=45.91.116.13:4153 , piece_id=GO5RMG74Z7VXFESGC6YGYYNOWPUUHQESTSFTCLCGJYFMRO7J6FMQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.388404386 (380h47m54.259487431s) [noise=true , address=145.79.227.48:4301 , piece_id=H2QPVXSJUSISVILZ56KA7B3K6WNPTO35TGCJMQUBKCGAEDLBRLHA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.388825608 (380h47m54.259908653s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.389159342 (380h47m54.260242378s) [noise=true , address=45.91.116.8:4047 , piece_id=UKQSJFOIWNPJPIRULJG2ABLRH2KGW4H6LXKBLTE63UEBEADJ2YYQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.389623908 (380h47m54.260706953s) [noise=true , address=145.79.227.39:4014 , piece_id=DIM6VYTUDPHYSI4OY7VEEZJB45SZWETSL6DIRIUTN4COPNEB3BNQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.390915798 (380h47m54.261998833s) [noise=true , address=145.79.227.46:4602 , piece_id=3F44IPNW6ADFR532FOCSF32RYCJF6UTI3BAYPHYZ32EAUN6IXT5A, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.391063709 (380h47m54.262146754s) [noise=true , address=145.79.227.43:4157 , piece_id=DW2XREE5F6IEPEYELDB3ZYARTXQOD6FH3DAH2ACMESP2RH4L3POA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.392035429 (380h47m54.263118464s) [noise=true , address=45.91.116.7:4009 , piece_id=6FEGEJDSB4NXVN5UDWYPENXYJZNBFP26KFAT7KIZEWUBDKEIIXUQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.392311255 (380h47m54.2633943s) [noise=true , address=145.79.227.42:4123 , piece_id=5DBX2SBAZV6NUPC7J7MSNW5OTXLQRWFLKYA6Y6IOHSD7K46QZBDA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.393169468 (380h47m54.264252513s) [noise=true , address=145.79.227.37:4566 , piece_id=BJD2RWGEMBFV3LANG5SKQP2M45UF3TH6LTJ5Q5JZ5H5RFRR6V55A, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.393483707 (380h47m54.264566742s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.394476878 (380h47m54.265559913s) [noise=true , address=77.90.31.17:4770 , piece_id=LSAQNVQV6ZEOKKBODN5FTVCGKPDN3D344NZ5HYNQA6TLUJBFDT3Q, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.394873383 (380h47m54.265956418s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.403422182 (380h47m54.274505217s) [noise=true , address=145.79.227.49:4339 , piece_id=6UIYCLJ2DXDKDLB6Q5TWUYENTTEASGE2VGCHF4HLK4JN2NQJO7HA, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:54.403874753 (380h47m54.274957788s) [noise=true , address=145.79.227.50:4374 , piece_id=YCR7XDSIAXRTUAKDMXR7PTJ73AJ7KPRBNICCQZLSSUV7AP5OVAHQ, seg_pos={0 0}, upload=334188]: Uploading piece... Dec 21 12:16:56.246383992 (380h47m56.117467027s) [upload=334188]: Next returned nil segment Dec 21 12:16:56.246409850 (380h47m56.117492885s) [upload=334188]: Waiting for error group managing segments... Dec 21 12:16:56.576072878 (380h47m56.447155933s) [noise=true , address=145.79.227.32:4835 , piece_id=DHQE5HL65NNZSIDCCIKASKM2BZUCCS5NXGW4VF7XWAARNCFUWOBQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.598667150 (380h47m56.469750185s) [noise=true , address=145.79.227.37:4566 , piece_id=BJD2RWGEMBFV3LANG5SKQP2M45UF3TH6LTJ5Q5JZ5H5RFRR6V55A, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.646458722 (380h47m56.517541767s) [noise=true , address=145.79.227.43:4157 , piece_id=DW2XREE5F6IEPEYELDB3ZYARTXQOD6FH3DAH2ACMESP2RH4L3POA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.679261544 (380h47m56.550344579s) [noise=true , address=77.90.31.17:4770 , piece_id=LSAQNVQV6ZEOKKBODN5FTVCGKPDN3D344NZ5HYNQA6TLUJBFDT3Q, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.688404533 (380h47m56.559487579s) [noise=true , address=45.91.116.8:4047 , piece_id=UKQSJFOIWNPJPIRULJG2ABLRH2KGW4H6LXKBLTE63UEBEADJ2YYQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.697132984 (380h47m56.568216019s) [noise=true , address=45.91.116.13:4153 , piece_id=GO5RMG74Z7VXFESGC6YGYYNOWPUUHQESTSFTCLCGJYFMRO7J6FMQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.698716369 (380h47m56.569799404s) [noise=true , address=45.91.116.7:4009 , piece_id=6FEGEJDSB4NXVN5UDWYPENXYJZNBFP26KFAT7KIZEWUBDKEIIXUQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.776653302 (380h47m56.647736337s) [noise=true , address=145.79.227.39:4014 , piece_id=DIM6VYTUDPHYSI4OY7VEEZJB45SZWETSL6DIRIUTN4COPNEB3BNQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.861714852 (380h47m56.732797887s) [noise=true , address=145.79.227.50:4374 , piece_id=YCR7XDSIAXRTUAKDMXR7PTJ73AJ7KPRBNICCQZLSSUV7AP5OVAHQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:56.870445466 (380h47m56.741528511s) [noise=true , address=145.79.227.46:4602 , piece_id=3F44IPNW6ADFR532FOCSF32RYCJF6UTI3BAYPHYZ32EAUN6IXT5A, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.278845830 (380h47m57.149928865s) [noise=true , address=145.79.227.49:4339 , piece_id=6UIYCLJ2DXDKDLB6Q5TWUYENTTEASGE2VGCHF4HLK4JN2NQJO7HA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.315489715 (380h47m57.18657276s) [noise=true , address=145.79.227.42:4123 , piece_id=5DBX2SBAZV6NUPC7J7MSNW5OTXLQRWFLKYA6Y6IOHSD7K46QZBDA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.388767035 (380h47m57.259850081s) [noise=true , address=145.79.227.48:4301 , piece_id=H2QPVXSJUSISVILZ56KA7B3K6WNPTO35TGCJMQUBKCGAEDLBRLHA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: <nil> Dec 21 12:16:57.388804847 (380h47m57.259887892s) [seg_pos={0 0}, upload=334188]: Segment reached optimal threshold of 13 pieces. Dec 21 12:16:57.389045496 (380h47m57.260128541s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389054282 (380h47m57.260137337s) [noise=true , address=145.79.227.51:4410 , piece_id=DGWOVKEVMKOPTILY5T6LKR4AJ3FXMMXWEXKABLON6QPGKIQ3ADJA, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389244826 (380h47m57.260327861s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389281535 (380h47m57.26036458s) [noise=true , address=145.79.227.8:4330 , piece_id=LPXQOHQIYQJPS3YVQTFQKX4547HZOR7VEJSWBBEN565T245LUVKQ, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389354815 (380h47m57.26043785s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Dec 21 12:16:57.389365345 (380h47m57.2604484s) [noise=true , address=145.79.227.47:4265 , piece_id=7BKRIHT3CAPUIA4KRRWE4S7HQMT4DG3QG2WBBBBH65INAC75XKZA, seg_pos={0 0}, upload=334188]: Long tail cancellation due to optimal threshold Dec 21 12:16:57.389391785 (380h47m57.26047483s) [seg_pos={0 0}, upload=334188]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:16:57.389478741 (380h47m57.260561786s) [upload=334188]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:16:57.389485083 (380h47m57.260568128s) [upload=334188]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:16:57.707610560 (380h47m57.578693595s) [upload=334188]: Done uploading segments... Dec 21 12:16:57.707629265 (380h47m57.5787123s) [upload=334188]: Upload finished. err: <nil> Duration: 30.003435092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X........................................XX.X..X.X..X.XXX.X.] duration: 25.620623121s trace: https://jaeger.dev.storj.io/trace/2a5aadb618cc14be FAIL (2025-12-21 12:16:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: 3JJD57OMHV7GW7BEXAA2NAHK37T4AQEMQP27NADHBDZT55MNCZRA) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: L5MEH4SOWZZZGZ72KULO7GP36TQYWJUKHVMZMWMXWXINXZ3SQ7SA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: QOR5OD5OOLSVDBXVZY65JIAVTO2GVB6M2LRNHTH525GK7ZEA3KKQ) context deadline exceeded ======LOGS====== Dec 21 12:15:22.518202237 (380h46m22.389285272s) [upload=334169]: Starting upload Dec 21 12:15:22.518279815 (380h46m22.38936285s) [upload=334169]: Done starting upload Dec 21 12:15:22.518417336 (380h46m22.389500382s) [upload=334169]: Uploading segments... Dec 21 12:15:22.518617668 (380h46m22.389700713s) [upload=334169]: Got next segment. Inline: false Dec 21 12:15:22.518666841 (380h46m22.389749876s) [upload=334169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:15:22.518673053 (380h46m22.389756108s) [upload=334169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:15:23.090297760 (380h46m22.961380805s) [seg_pos={0 0}, upload=334169]: Begin upload segment... Dec 21 12:15:23.090359227 (380h46m22.961442262s) [seg_pos={0 0}, upload=334169]: Done begin upload segment. Dec 21 12:15:23.090395446 (380h46m22.961478492s) [noise=true , address=145.79.227.2:4115 , piece_id=UKSKE6I57H6IHME2UGMCNWBATE3646TKB22H6GLWXIN47S6PMFXQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.090472563 (380h46m22.961555598s) [noise=true , address=145.79.227.45:4231 , piece_id=7YZZUQJCB6QQ2XNGB4ZNBNT3RPOZHAI6BUWX56H36QYXNTFPIUSA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.091113533 (380h46m22.962196578s) [noise=true , address=145.79.227.34:4907 , piece_id=4M67SWCBGKF3WTYEBLCZGB4X5GJGGH7R6W2HRGCXWU7ULL3BIFPA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.091630818 (380h46m22.962713863s) [noise=true , address=145.79.227.26:4655 , piece_id=WS76FGYGDKODX5HJ3LCLTKCMOX7HMN5Q5EVIQOSU2KSKT2POWX5A, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.092058001 (380h46m22.963141037s) [noise=true , address=145.79.227.52:4447 , piece_id=GFOCXR5TY3RMYBZSJO4QVXN7H3M5EE7WADIJ2UNTT47FZFWM5YPQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.092499462 (380h46m22.963582497s) [noise=true , address=145.79.227.43:4159 , piece_id=6CTCG4D2262Y57FQUEGAZDVSREDSD5YVEPROE3OJIYG2YDGYILVA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.092907900 (380h46m22.963990946s) [noise=true , address=145.79.227.37:4567 , piece_id=3JJD57OMHV7GW7BEXAA2NAHK37T4AQEMQP27NADHBDZT55MNCZRA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.093357046 (380h46m22.964440091s) [noise=true , address=145.79.227.51:4409 , piece_id=RQ3SMRGEUI6EQ3H3M5DYOO5MK7BXEIBOATDWONNZ6BYDIJJVQFCQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.093980703 (380h46m22.965063738s) [noise=true , address=145.79.227.4:4186 , piece_id=43CSNDEQZYRJ2UICKN4BD66QLPP4L2MW4HKQW2HMXCFOTUXDXY4A, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.094470496 (380h46m22.965553531s) [noise=true , address=145.79.227.50:4375 , piece_id=L5MEH4SOWZZZGZ72KULO7GP36TQYWJUKHVMZMWMXWXINXZ3SQ7SA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.094535719 (380h46m22.965618754s) [noise=true , address=145.79.227.17:4972 , piece_id=3CKY2GMONA23PV5EPIVGDB6SPHQBODJCUTSOKQW7TRDMVXNMVW2Q, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.095871303 (380h46m22.966954348s) [noise=true , address=45.91.116.12:4227 , piece_id=5H75CTHH6JAZORGF5ACVBPUJN4NHPKAGEXEKGSEHDFJCXW6UHY6Q, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.097960480 (380h46m22.969043525s) [noise=true , address=45.91.116.14:4189 , piece_id=W76LS5RIVCQZY4SBQOCOXGDE4MKA3NTAD3WKAHTTTGABCJ2SPKTA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.099913820 (380h46m22.970996865s) [noise=true , address=77.90.31.13:4629 , piece_id=3IR66SMNFTMQA2JSHUGXQHEYIMU4WNU2OPWKTPO6W6QMWF7HC7VQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.101554224 (380h46m22.972637259s) [noise=true , address=45.91.116.4:4940 , piece_id=ARF6ZZ2WLIDKXDU37H74G3QYRW6FZUUKDKASKL2EXXBCEFBRC3YQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.103146095 (380h46m22.97422914s) [noise=true , address=145.79.227.20:4437 , piece_id=QOR5OD5OOLSVDBXVZY65JIAVTO2GVB6M2LRNHTH525GK7ZEA3KKQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:24.946573016 (380h46m24.817656051s) [upload=334169]: Next returned nil segment Dec 21 12:15:24.946592223 (380h46m24.817675268s) [upload=334169]: Waiting for error group managing segments... Dec 21 12:15:25.252017919 (380h46m25.123100954s) [noise=true , address=145.79.227.45:4231 , piece_id=7YZZUQJCB6QQ2XNGB4ZNBNT3RPOZHAI6BUWX56H36QYXNTFPIUSA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.263754917 (380h46m25.134837952s) [noise=true , address=145.79.227.2:4115 , piece_id=UKSKE6I57H6IHME2UGMCNWBATE3646TKB22H6GLWXIN47S6PMFXQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.267733974 (380h46m25.138817019s) [noise=true , address=145.79.227.43:4159 , piece_id=6CTCG4D2262Y57FQUEGAZDVSREDSD5YVEPROE3OJIYG2YDGYILVA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.302255478 (380h46m25.173338513s) [noise=true , address=145.79.227.52:4447 , piece_id=GFOCXR5TY3RMYBZSJO4QVXN7H3M5EE7WADIJ2UNTT47FZFWM5YPQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.302439919 (380h46m25.173522954s) [noise=true , address=145.79.227.17:4972 , piece_id=3CKY2GMONA23PV5EPIVGDB6SPHQBODJCUTSOKQW7TRDMVXNMVW2Q, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.349666746 (380h46m25.220749781s) [noise=true , address=145.79.227.20:4437 , piece_id=QOR5OD5OOLSVDBXVZY65JIAVTO2GVB6M2LRNHTH525GK7ZEA3KKQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.372231603 (380h46m25.243314638s) [noise=true , address=77.90.31.13:4629 , piece_id=3IR66SMNFTMQA2JSHUGXQHEYIMU4WNU2OPWKTPO6W6QMWF7HC7VQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.395317852 (380h46m25.266400887s) [noise=true , address=45.91.116.12:4227 , piece_id=5H75CTHH6JAZORGF5ACVBPUJN4NHPKAGEXEKGSEHDFJCXW6UHY6Q, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.497726544 (380h46m25.368809579s) [noise=true , address=145.79.227.37:4567 , piece_id=3JJD57OMHV7GW7BEXAA2NAHK37T4AQEMQP27NADHBDZT55MNCZRA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.607777987 (380h46m25.478861032s) [noise=true , address=145.79.227.4:4186 , piece_id=43CSNDEQZYRJ2UICKN4BD66QLPP4L2MW4HKQW2HMXCFOTUXDXY4A, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.953072368 (380h46m25.824155403s) [noise=true , address=145.79.227.34:4907 , piece_id=4M67SWCBGKF3WTYEBLCZGB4X5GJGGH7R6W2HRGCXWU7ULL3BIFPA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:26.113378574 (380h46m25.984461619s) [noise=true , address=145.79.227.50:4375 , piece_id=L5MEH4SOWZZZGZ72KULO7GP36TQYWJUKHVMZMWMXWXINXZ3SQ7SA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:26.290782826 (380h46m26.161865881s) [noise=true , address=145.79.227.26:4655 , piece_id=WS76FGYGDKODX5HJ3LCLTKCMOX7HMN5Q5EVIQOSU2KSKT2POWX5A, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:26.290806702 (380h46m26.161889737s) [seg_pos={0 0}, upload=334169]: Segment reached optimal threshold of 13 pieces. Dec 21 12:15:26.291229697 (380h46m26.162312742s) [noise=true , address=45.91.116.14:4189 , piece_id=W76LS5RIVCQZY4SBQOCOXGDE4MKA3NTAD3WKAHTTTGABCJ2SPKTA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 21 12:15:26.291245416 (380h46m26.162328462s) [noise=true , address=45.91.116.14:4189 , piece_id=W76LS5RIVCQZY4SBQOCOXGDE4MKA3NTAD3WKAHTTTGABCJ2SPKTA, seg_pos={0 0}, upload=334169]: Long tail cancellation due to optimal threshold Dec 21 12:15:26.291517745 (380h46m26.16260078s) [noise=true , address=45.91.116.4:4940 , piece_id=ARF6ZZ2WLIDKXDU37H74G3QYRW6FZUUKDKASKL2EXXBCEFBRC3YQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 12:15:26.291525760 (380h46m26.162608795s) [noise=true , address=45.91.116.4:4940 , piece_id=ARF6ZZ2WLIDKXDU37H74G3QYRW6FZUUKDKASKL2EXXBCEFBRC3YQ, seg_pos={0 0}, upload=334169]: Long tail cancellation due to optimal threshold Dec 21 12:15:26.291629408 (380h46m26.162712473s) [noise=true , address=145.79.227.51:4409 , piece_id=RQ3SMRGEUI6EQ3H3M5DYOO5MK7BXEIBOATDWONNZ6BYDIJJVQFCQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 21 12:15:26.291641391 (380h46m26.162724426s) [noise=true , address=145.79.227.51:4409 , piece_id=RQ3SMRGEUI6EQ3H3M5DYOO5MK7BXEIBOATDWONNZ6BYDIJJVQFCQ, seg_pos={0 0}, upload=334169]: Long tail cancellation due to optimal threshold Dec 21 12:15:26.291669685 (380h46m26.16275273s) [seg_pos={0 0}, upload=334169]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:15:26.291763253 (380h46m26.162846298s) [upload=334169]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:15:26.291770387 (380h46m26.162853432s) [upload=334169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:15:26.606723584 (380h46m26.477806619s) [upload=334169]: Done uploading segments... Dec 21 12:15:26.606745155 (380h46m26.4778282s) [upload=334169]: Upload finished. err: <nil> Duration: 25.620643309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X........................................XX.X..X.X..X.XXX.X] duration: 30.002380599s trace: https://jaeger.dev.storj.io/trace/2a5aadb618cc14be FAIL (2025-12-21 12:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, Piece ID: 3JJD57OMHV7GW7BEXAA2NAHK37T4AQEMQP27NADHBDZT55MNCZRA) context deadline exceeded; piecestore: (Node ID: 121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, Piece ID: L5MEH4SOWZZZGZ72KULO7GP36TQYWJUKHVMZMWMXWXINXZ3SQ7SA) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: QOR5OD5OOLSVDBXVZY65JIAVTO2GVB6M2LRNHTH525GK7ZEA3KKQ) context deadline exceeded ======LOGS====== Dec 21 12:15:22.518202237 (380h46m22.389285272s) [upload=334169]: Starting upload Dec 21 12:15:22.518279815 (380h46m22.38936285s) [upload=334169]: Done starting upload Dec 21 12:15:22.518417336 (380h46m22.389500382s) [upload=334169]: Uploading segments... Dec 21 12:15:22.518617668 (380h46m22.389700713s) [upload=334169]: Got next segment. Inline: false Dec 21 12:15:22.518666841 (380h46m22.389749876s) [upload=334169]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:15:22.518673053 (380h46m22.389756108s) [upload=334169]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:15:23.090297760 (380h46m22.961380805s) [seg_pos={0 0}, upload=334169]: Begin upload segment... Dec 21 12:15:23.090359227 (380h46m22.961442262s) [seg_pos={0 0}, upload=334169]: Done begin upload segment. Dec 21 12:15:23.090395446 (380h46m22.961478492s) [noise=true , address=145.79.227.2:4115 , piece_id=UKSKE6I57H6IHME2UGMCNWBATE3646TKB22H6GLWXIN47S6PMFXQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.090472563 (380h46m22.961555598s) [noise=true , address=145.79.227.45:4231 , piece_id=7YZZUQJCB6QQ2XNGB4ZNBNT3RPOZHAI6BUWX56H36QYXNTFPIUSA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.091113533 (380h46m22.962196578s) [noise=true , address=145.79.227.34:4907 , piece_id=4M67SWCBGKF3WTYEBLCZGB4X5GJGGH7R6W2HRGCXWU7ULL3BIFPA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.091630818 (380h46m22.962713863s) [noise=true , address=145.79.227.26:4655 , piece_id=WS76FGYGDKODX5HJ3LCLTKCMOX7HMN5Q5EVIQOSU2KSKT2POWX5A, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.092058001 (380h46m22.963141037s) [noise=true , address=145.79.227.52:4447 , piece_id=GFOCXR5TY3RMYBZSJO4QVXN7H3M5EE7WADIJ2UNTT47FZFWM5YPQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.092499462 (380h46m22.963582497s) [noise=true , address=145.79.227.43:4159 , piece_id=6CTCG4D2262Y57FQUEGAZDVSREDSD5YVEPROE3OJIYG2YDGYILVA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.092907900 (380h46m22.963990946s) [noise=true , address=145.79.227.37:4567 , piece_id=3JJD57OMHV7GW7BEXAA2NAHK37T4AQEMQP27NADHBDZT55MNCZRA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.093357046 (380h46m22.964440091s) [noise=true , address=145.79.227.51:4409 , piece_id=RQ3SMRGEUI6EQ3H3M5DYOO5MK7BXEIBOATDWONNZ6BYDIJJVQFCQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.093980703 (380h46m22.965063738s) [noise=true , address=145.79.227.4:4186 , piece_id=43CSNDEQZYRJ2UICKN4BD66QLPP4L2MW4HKQW2HMXCFOTUXDXY4A, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.094470496 (380h46m22.965553531s) [noise=true , address=145.79.227.50:4375 , piece_id=L5MEH4SOWZZZGZ72KULO7GP36TQYWJUKHVMZMWMXWXINXZ3SQ7SA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.094535719 (380h46m22.965618754s) [noise=true , address=145.79.227.17:4972 , piece_id=3CKY2GMONA23PV5EPIVGDB6SPHQBODJCUTSOKQW7TRDMVXNMVW2Q, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.095871303 (380h46m22.966954348s) [noise=true , address=45.91.116.12:4227 , piece_id=5H75CTHH6JAZORGF5ACVBPUJN4NHPKAGEXEKGSEHDFJCXW6UHY6Q, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.097960480 (380h46m22.969043525s) [noise=true , address=45.91.116.14:4189 , piece_id=W76LS5RIVCQZY4SBQOCOXGDE4MKA3NTAD3WKAHTTTGABCJ2SPKTA, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.099913820 (380h46m22.970996865s) [noise=true , address=77.90.31.13:4629 , piece_id=3IR66SMNFTMQA2JSHUGXQHEYIMU4WNU2OPWKTPO6W6QMWF7HC7VQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.101554224 (380h46m22.972637259s) [noise=true , address=45.91.116.4:4940 , piece_id=ARF6ZZ2WLIDKXDU37H74G3QYRW6FZUUKDKASKL2EXXBCEFBRC3YQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:23.103146095 (380h46m22.97422914s) [noise=true , address=145.79.227.20:4437 , piece_id=QOR5OD5OOLSVDBXVZY65JIAVTO2GVB6M2LRNHTH525GK7ZEA3KKQ, seg_pos={0 0}, upload=334169]: Uploading piece... Dec 21 12:15:24.946573016 (380h46m24.817656051s) [upload=334169]: Next returned nil segment Dec 21 12:15:24.946592223 (380h46m24.817675268s) [upload=334169]: Waiting for error group managing segments... Dec 21 12:15:25.252017919 (380h46m25.123100954s) [noise=true , address=145.79.227.45:4231 , piece_id=7YZZUQJCB6QQ2XNGB4ZNBNT3RPOZHAI6BUWX56H36QYXNTFPIUSA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.263754917 (380h46m25.134837952s) [noise=true , address=145.79.227.2:4115 , piece_id=UKSKE6I57H6IHME2UGMCNWBATE3646TKB22H6GLWXIN47S6PMFXQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.267733974 (380h46m25.138817019s) [noise=true , address=145.79.227.43:4159 , piece_id=6CTCG4D2262Y57FQUEGAZDVSREDSD5YVEPROE3OJIYG2YDGYILVA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.302255478 (380h46m25.173338513s) [noise=true , address=145.79.227.52:4447 , piece_id=GFOCXR5TY3RMYBZSJO4QVXN7H3M5EE7WADIJ2UNTT47FZFWM5YPQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.302439919 (380h46m25.173522954s) [noise=true , address=145.79.227.17:4972 , piece_id=3CKY2GMONA23PV5EPIVGDB6SPHQBODJCUTSOKQW7TRDMVXNMVW2Q, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.349666746 (380h46m25.220749781s) [noise=true , address=145.79.227.20:4437 , piece_id=QOR5OD5OOLSVDBXVZY65JIAVTO2GVB6M2LRNHTH525GK7ZEA3KKQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.372231603 (380h46m25.243314638s) [noise=true , address=77.90.31.13:4629 , piece_id=3IR66SMNFTMQA2JSHUGXQHEYIMU4WNU2OPWKTPO6W6QMWF7HC7VQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.395317852 (380h46m25.266400887s) [noise=true , address=45.91.116.12:4227 , piece_id=5H75CTHH6JAZORGF5ACVBPUJN4NHPKAGEXEKGSEHDFJCXW6UHY6Q, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.497726544 (380h46m25.368809579s) [noise=true , address=145.79.227.37:4567 , piece_id=3JJD57OMHV7GW7BEXAA2NAHK37T4AQEMQP27NADHBDZT55MNCZRA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.607777987 (380h46m25.478861032s) [noise=true , address=145.79.227.4:4186 , piece_id=43CSNDEQZYRJ2UICKN4BD66QLPP4L2MW4HKQW2HMXCFOTUXDXY4A, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:25.953072368 (380h46m25.824155403s) [noise=true , address=145.79.227.34:4907 , piece_id=4M67SWCBGKF3WTYEBLCZGB4X5GJGGH7R6W2HRGCXWU7ULL3BIFPA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:26.113378574 (380h46m25.984461619s) [noise=true , address=145.79.227.50:4375 , piece_id=L5MEH4SOWZZZGZ72KULO7GP36TQYWJUKHVMZMWMXWXINXZ3SQ7SA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:26.290782826 (380h46m26.161865881s) [noise=true , address=145.79.227.26:4655 , piece_id=WS76FGYGDKODX5HJ3LCLTKCMOX7HMN5Q5EVIQOSU2KSKT2POWX5A, seg_pos={0 0}, upload=334169]: Done uploading piece. err: <nil> Dec 21 12:15:26.290806702 (380h46m26.161889737s) [seg_pos={0 0}, upload=334169]: Segment reached optimal threshold of 13 pieces. Dec 21 12:15:26.291229697 (380h46m26.162312742s) [noise=true , address=45.91.116.14:4189 , piece_id=W76LS5RIVCQZY4SBQOCOXGDE4MKA3NTAD3WKAHTTTGABCJ2SPKTA, seg_pos={0 0}, upload=334169]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 21 12:15:26.291245416 (380h46m26.162328462s) [noise=true , address=45.91.116.14:4189 , piece_id=W76LS5RIVCQZY4SBQOCOXGDE4MKA3NTAD3WKAHTTTGABCJ2SPKTA, seg_pos={0 0}, upload=334169]: Long tail cancellation due to optimal threshold Dec 21 12:15:26.291517745 (380h46m26.16260078s) [noise=true , address=45.91.116.4:4940 , piece_id=ARF6ZZ2WLIDKXDU37H74G3QYRW6FZUUKDKASKL2EXXBCEFBRC3YQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 12:15:26.291525760 (380h46m26.162608795s) [noise=true , address=45.91.116.4:4940 , piece_id=ARF6ZZ2WLIDKXDU37H74G3QYRW6FZUUKDKASKL2EXXBCEFBRC3YQ, seg_pos={0 0}, upload=334169]: Long tail cancellation due to optimal threshold Dec 21 12:15:26.291629408 (380h46m26.162712473s) [noise=true , address=145.79.227.51:4409 , piece_id=RQ3SMRGEUI6EQ3H3M5DYOO5MK7BXEIBOATDWONNZ6BYDIJJVQFCQ, seg_pos={0 0}, upload=334169]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 21 12:15:26.291641391 (380h46m26.162724426s) [noise=true , address=145.79.227.51:4409 , piece_id=RQ3SMRGEUI6EQ3H3M5DYOO5MK7BXEIBOATDWONNZ6BYDIJJVQFCQ, seg_pos={0 0}, upload=334169]: Long tail cancellation due to optimal threshold Dec 21 12:15:26.291669685 (380h46m26.16275273s) [seg_pos={0 0}, upload=334169]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:15:26.291763253 (380h46m26.162846298s) [upload=334169]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:15:26.291770387 (380h46m26.162853432s) [upload=334169]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:15:26.606723584 (380h46m26.477806619s) [upload=334169]: Done uploading segments... Dec 21 12:15:26.606745155 (380h46m26.4778282s) [upload=334169]: Upload finished. err: <nil> Duration: 30.002404194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X........................................XX.X..X.X..X.XXX.] duration: 28.478343858s trace: https://jaeger.dev.storj.io/trace/207aa32871487aaf FAIL (2025-12-21 12:14:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: 6COIQ3HVSY7QT3UGW7FSLAWRKELQZKVAEWCTBAPH37JYFWWADYUQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: RYKJRPSGBJED4J25YP7JS4R42HKA474563X6ZKUBCP7J5MGAWTPQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: 3HU54QLVMYN7LSYLNKNAVTFYUOM6LIGOJ2A6S5EYMD7DBLHMCWRQ) context deadline exceeded; piecestore: (Node ID: 12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, Piece ID: 4TEDJLJQK7FLAGKHR2OQYNCCYGNAPO2YQJT2UNFNYZIPXMVV6A7A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: NNSDYGUPN3QVNZ63QRLERD3XD4SYETHL44YGA6435HKXFMAH5KLA) context deadline exceeded ======LOGS====== Dec 21 12:14:00.064641662 (380h44m59.935724698s) [upload=334148]: Starting upload Dec 21 12:14:00.064722586 (380h44m59.935805631s) [upload=334148]: Done starting upload Dec 21 12:14:00.064814462 (380h44m59.935897497s) [upload=334148]: Uploading segments... Dec 21 12:14:00.065075087 (380h44m59.936158132s) [upload=334148]: Got next segment. Inline: false Dec 21 12:14:00.065193723 (380h44m59.936276758s) [upload=334148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:14:00.065200497 (380h44m59.936283532s) [upload=334148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:14:00.306218941 (380h45m0.177301976s) [seg_pos={0 0}, upload=334148]: Begin upload segment... Dec 21 12:14:00.306270959 (380h45m0.177354004s) [seg_pos={0 0}, upload=334148]: Done begin upload segment. Dec 21 12:14:00.306316576 (380h45m0.177399621s) [noise=true , address=145.79.227.41:4087 , piece_id=TN4YDWBFCZFTZD7DG5RES7UB3MXSUWZRML5ME3J4AO2XM7IRF2YA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.306411827 (380h45m0.177494862s) [noise=true , address=145.79.227.14:4840 , piece_id=MZ3ECARQPDOYZPZZVMJWQFYX4XLAVCYAE62FPFBT4JT6HHF2W5PQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.306979669 (380h45m0.178062714s) [noise=true , address=145.79.227.42:4123 , piece_id=GASMH7TF53DYCQD77KKKZL7G4YRKV2ICAIQUXDW2I7XQXIKLZTUA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.307512844 (380h45m0.178595889s) [noise=true , address=145.79.227.32:4835 , piece_id=QNV2QF3NSLYDWWWPFEFXOK5IVPS2D4ZGNUGUNOTHWAWNO7RIRCNQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.307668530 (380h45m0.178751565s) [noise=true , address=145.79.227.10:4403 , piece_id=ADNUV4N3MEUYPYJBMXFURSR4JNL6F5HYMS4HFUQCGMWR2N2252TA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.308387349 (380h45m0.179470384s) [noise=true , address=145.79.227.8:4330 , piece_id=NNSDYGUPN3QVNZ63QRLERD3XD4SYETHL44YGA6435HKXFMAH5KLA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.308711246 (380h45m0.179794291s) [noise=true , address=145.79.227.29:4942 , piece_id=EHUOHUW4T37BC4QLP5T37JLV35MDMEWF3WQYDDK4IL7FWDDBWFNQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.308926025 (380h45m0.18000907s) [noise=true , address=145.79.227.38:4977 , piece_id=IUDZFALFSX5QE26GVWMXNP6GP35W3C24HNIHADQPDL6OK2GEUHGA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309217069 (380h45m0.180300115s) [noise=true , address=145.79.227.36:4519 , piece_id=YVLIPA2GWTENE37FIHWRD4XOF3MBNXJUNLHLG6GBEAN56E5SZOXQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309239091 (380h45m0.180322136s) [noise=true , address=145.79.227.49:4337 , piece_id=6COIQ3HVSY7QT3UGW7FSLAWRKELQZKVAEWCTBAPH37JYFWWADYUQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309543601 (380h45m0.180626646s) [noise=true , address=145.79.227.39:4014 , piece_id=RYKJRPSGBJED4J25YP7JS4R42HKA474563X6ZKUBCP7J5MGAWTPQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309691492 (380h45m0.180774537s) [noise=true , address=145.79.227.51:4410 , piece_id=3HU54QLVMYN7LSYLNKNAVTFYUOM6LIGOJ2A6S5EYMD7DBLHMCWRQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309997295 (380h45m0.18108033s) [noise=true , address=45.91.116.7:4012 , piece_id=4ZUTJ5QSPSGYLX7ZO34GU3WF3URE3TOEQPM7Q6ISSZVWZPNBZ4XQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.310238935 (380h45m0.18132198s) [noise=true , address=145.79.227.26:4653 , piece_id=4TEDJLJQK7FLAGKHR2OQYNCCYGNAPO2YQJT2UNFNYZIPXMVV6A7A, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.311542468 (380h45m0.182625503s) [noise=true , address=45.91.116.4:4937 , piece_id=6ANWFRY3H7T4ETX5VT2FQNKNDSIY2ICSDYTKIZKX6TVM4AXUXIDA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.314963341 (380h45m0.186046376s) [noise=true , address=145.79.227.37:4565 , piece_id=RUIB67LFS5PSVAQQKL7XMI5MBXEVA42X52LTXCPPSRPCGUYWCDGQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:02.142871366 (380h45m2.013954411s) [upload=334148]: Next returned nil segment Dec 21 12:14:02.142898999 (380h45m2.013982044s) [upload=334148]: Waiting for error group managing segments... Dec 21 12:14:02.452330661 (380h45m2.323413706s) [noise=true , address=145.79.227.42:4123 , piece_id=GASMH7TF53DYCQD77KKKZL7G4YRKV2ICAIQUXDW2I7XQXIKLZTUA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.471141471 (380h45m2.342224516s) [noise=true , address=145.79.227.32:4835 , piece_id=QNV2QF3NSLYDWWWPFEFXOK5IVPS2D4ZGNUGUNOTHWAWNO7RIRCNQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.475332521 (380h45m2.346415566s) [noise=true , address=145.79.227.26:4653 , piece_id=4TEDJLJQK7FLAGKHR2OQYNCCYGNAPO2YQJT2UNFNYZIPXMVV6A7A, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.489574809 (380h45m2.360657855s) [noise=true , address=145.79.227.37:4565 , piece_id=RUIB67LFS5PSVAQQKL7XMI5MBXEVA42X52LTXCPPSRPCGUYWCDGQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.495733106 (380h45m2.366816151s) [noise=true , address=145.79.227.49:4337 , piece_id=6COIQ3HVSY7QT3UGW7FSLAWRKELQZKVAEWCTBAPH37JYFWWADYUQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.540052607 (380h45m2.411135652s) [noise=true , address=145.79.227.51:4410 , piece_id=3HU54QLVMYN7LSYLNKNAVTFYUOM6LIGOJ2A6S5EYMD7DBLHMCWRQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.545014355 (380h45m2.4160974s) [noise=true , address=145.79.227.10:4403 , piece_id=ADNUV4N3MEUYPYJBMXFURSR4JNL6F5HYMS4HFUQCGMWR2N2252TA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.554598926 (380h45m2.425681971s) [noise=true , address=145.79.227.39:4014 , piece_id=RYKJRPSGBJED4J25YP7JS4R42HKA474563X6ZKUBCP7J5MGAWTPQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.560517024 (380h45m2.431600069s) [noise=true , address=145.79.227.38:4977 , piece_id=IUDZFALFSX5QE26GVWMXNP6GP35W3C24HNIHADQPDL6OK2GEUHGA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.593460894 (380h45m2.464543939s) [noise=true , address=45.91.116.4:4937 , piece_id=6ANWFRY3H7T4ETX5VT2FQNKNDSIY2ICSDYTKIZKX6TVM4AXUXIDA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.596454945 (380h45m2.46753799s) [noise=true , address=45.91.116.7:4012 , piece_id=4ZUTJ5QSPSGYLX7ZO34GU3WF3URE3TOEQPM7Q6ISSZVWZPNBZ4XQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.829578497 (380h45m2.700661542s) [noise=true , address=145.79.227.8:4330 , piece_id=NNSDYGUPN3QVNZ63QRLERD3XD4SYETHL44YGA6435HKXFMAH5KLA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:03.167529816 (380h45m3.038612861s) [noise=true , address=145.79.227.41:4087 , piece_id=TN4YDWBFCZFTZD7DG5RES7UB3MXSUWZRML5ME3J4AO2XM7IRF2YA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:03.167547389 (380h45m3.038630424s) [seg_pos={0 0}, upload=334148]: Segment reached optimal threshold of 13 pieces. Dec 21 12:14:03.167724857 (380h45m3.038807902s) [noise=true , address=145.79.227.29:4942 , piece_id=EHUOHUW4T37BC4QLP5T37JLV35MDMEWF3WQYDDK4IL7FWDDBWFNQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 21 12:14:03.167736028 (380h45m3.038819073s) [noise=true , address=145.79.227.29:4942 , piece_id=EHUOHUW4T37BC4QLP5T37JLV35MDMEWF3WQYDDK4IL7FWDDBWFNQ, seg_pos={0 0}, upload=334148]: Long tail cancellation due to optimal threshold Dec 21 12:14:03.167922023 (380h45m3.039005058s) [noise=true , address=145.79.227.14:4840 , piece_id=MZ3ECARQPDOYZPZZVMJWQFYX4XLAVCYAE62FPFBT4JT6HHF2W5PQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 21 12:14:03.167938504 (380h45m3.039021549s) [noise=true , address=145.79.227.14:4840 , piece_id=MZ3ECARQPDOYZPZZVMJWQFYX4XLAVCYAE62FPFBT4JT6HHF2W5PQ, seg_pos={0 0}, upload=334148]: Long tail cancellation due to optimal threshold Dec 21 12:14:03.168068993 (380h45m3.039152028s) [noise=true , address=145.79.227.36:4519 , piece_id=YVLIPA2GWTENE37FIHWRD4XOF3MBNXJUNLHLG6GBEAN56E5SZOXQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 21 12:14:03.168076807 (380h45m3.039159852s) [noise=true , address=145.79.227.36:4519 , piece_id=YVLIPA2GWTENE37FIHWRD4XOF3MBNXJUNLHLG6GBEAN56E5SZOXQ, seg_pos={0 0}, upload=334148]: Long tail cancellation due to optimal threshold Dec 21 12:14:03.168096385 (380h45m3.03917943s) [seg_pos={0 0}, upload=334148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:14:03.168166759 (380h45m3.039249804s) [upload=334148]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:14:03.168172179 (380h45m3.039255224s) [upload=334148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:14:03.460259967 (380h45m3.331343022s) [upload=334148]: Done uploading segments... Dec 21 12:14:03.460287389 (380h45m3.331370434s) [upload=334148]: Upload finished. err: <nil> Duration: 28.478475549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X........................................XX.X..X.X..X.XXX] duration: 30.002928028s trace: https://jaeger.dev.storj.io/trace/207aa32871487aaf FAIL (2025-12-21 12:14:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: 6COIQ3HVSY7QT3UGW7FSLAWRKELQZKVAEWCTBAPH37JYFWWADYUQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: RYKJRPSGBJED4J25YP7JS4R42HKA474563X6ZKUBCP7J5MGAWTPQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: 3HU54QLVMYN7LSYLNKNAVTFYUOM6LIGOJ2A6S5EYMD7DBLHMCWRQ) context deadline exceeded; piecestore: (Node ID: 12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, Piece ID: 4TEDJLJQK7FLAGKHR2OQYNCCYGNAPO2YQJT2UNFNYZIPXMVV6A7A) context deadline exceeded; piecestore: (Node ID: 125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, Piece ID: NNSDYGUPN3QVNZ63QRLERD3XD4SYETHL44YGA6435HKXFMAH5KLA) context deadline exceeded ======LOGS====== Dec 21 12:14:00.064641662 (380h44m59.935724698s) [upload=334148]: Starting upload Dec 21 12:14:00.064722586 (380h44m59.935805631s) [upload=334148]: Done starting upload Dec 21 12:14:00.064814462 (380h44m59.935897497s) [upload=334148]: Uploading segments... Dec 21 12:14:00.065075087 (380h44m59.936158132s) [upload=334148]: Got next segment. Inline: false Dec 21 12:14:00.065193723 (380h44m59.936276758s) [upload=334148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:14:00.065200497 (380h44m59.936283532s) [upload=334148]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:14:00.306218941 (380h45m0.177301976s) [seg_pos={0 0}, upload=334148]: Begin upload segment... Dec 21 12:14:00.306270959 (380h45m0.177354004s) [seg_pos={0 0}, upload=334148]: Done begin upload segment. Dec 21 12:14:00.306316576 (380h45m0.177399621s) [noise=true , address=145.79.227.41:4087 , piece_id=TN4YDWBFCZFTZD7DG5RES7UB3MXSUWZRML5ME3J4AO2XM7IRF2YA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.306411827 (380h45m0.177494862s) [noise=true , address=145.79.227.14:4840 , piece_id=MZ3ECARQPDOYZPZZVMJWQFYX4XLAVCYAE62FPFBT4JT6HHF2W5PQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.306979669 (380h45m0.178062714s) [noise=true , address=145.79.227.42:4123 , piece_id=GASMH7TF53DYCQD77KKKZL7G4YRKV2ICAIQUXDW2I7XQXIKLZTUA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.307512844 (380h45m0.178595889s) [noise=true , address=145.79.227.32:4835 , piece_id=QNV2QF3NSLYDWWWPFEFXOK5IVPS2D4ZGNUGUNOTHWAWNO7RIRCNQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.307668530 (380h45m0.178751565s) [noise=true , address=145.79.227.10:4403 , piece_id=ADNUV4N3MEUYPYJBMXFURSR4JNL6F5HYMS4HFUQCGMWR2N2252TA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.308387349 (380h45m0.179470384s) [noise=true , address=145.79.227.8:4330 , piece_id=NNSDYGUPN3QVNZ63QRLERD3XD4SYETHL44YGA6435HKXFMAH5KLA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.308711246 (380h45m0.179794291s) [noise=true , address=145.79.227.29:4942 , piece_id=EHUOHUW4T37BC4QLP5T37JLV35MDMEWF3WQYDDK4IL7FWDDBWFNQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.308926025 (380h45m0.18000907s) [noise=true , address=145.79.227.38:4977 , piece_id=IUDZFALFSX5QE26GVWMXNP6GP35W3C24HNIHADQPDL6OK2GEUHGA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309217069 (380h45m0.180300115s) [noise=true , address=145.79.227.36:4519 , piece_id=YVLIPA2GWTENE37FIHWRD4XOF3MBNXJUNLHLG6GBEAN56E5SZOXQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309239091 (380h45m0.180322136s) [noise=true , address=145.79.227.49:4337 , piece_id=6COIQ3HVSY7QT3UGW7FSLAWRKELQZKVAEWCTBAPH37JYFWWADYUQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309543601 (380h45m0.180626646s) [noise=true , address=145.79.227.39:4014 , piece_id=RYKJRPSGBJED4J25YP7JS4R42HKA474563X6ZKUBCP7J5MGAWTPQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309691492 (380h45m0.180774537s) [noise=true , address=145.79.227.51:4410 , piece_id=3HU54QLVMYN7LSYLNKNAVTFYUOM6LIGOJ2A6S5EYMD7DBLHMCWRQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.309997295 (380h45m0.18108033s) [noise=true , address=45.91.116.7:4012 , piece_id=4ZUTJ5QSPSGYLX7ZO34GU3WF3URE3TOEQPM7Q6ISSZVWZPNBZ4XQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.310238935 (380h45m0.18132198s) [noise=true , address=145.79.227.26:4653 , piece_id=4TEDJLJQK7FLAGKHR2OQYNCCYGNAPO2YQJT2UNFNYZIPXMVV6A7A, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.311542468 (380h45m0.182625503s) [noise=true , address=45.91.116.4:4937 , piece_id=6ANWFRY3H7T4ETX5VT2FQNKNDSIY2ICSDYTKIZKX6TVM4AXUXIDA, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:00.314963341 (380h45m0.186046376s) [noise=true , address=145.79.227.37:4565 , piece_id=RUIB67LFS5PSVAQQKL7XMI5MBXEVA42X52LTXCPPSRPCGUYWCDGQ, seg_pos={0 0}, upload=334148]: Uploading piece... Dec 21 12:14:02.142871366 (380h45m2.013954411s) [upload=334148]: Next returned nil segment Dec 21 12:14:02.142898999 (380h45m2.013982044s) [upload=334148]: Waiting for error group managing segments... Dec 21 12:14:02.452330661 (380h45m2.323413706s) [noise=true , address=145.79.227.42:4123 , piece_id=GASMH7TF53DYCQD77KKKZL7G4YRKV2ICAIQUXDW2I7XQXIKLZTUA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.471141471 (380h45m2.342224516s) [noise=true , address=145.79.227.32:4835 , piece_id=QNV2QF3NSLYDWWWPFEFXOK5IVPS2D4ZGNUGUNOTHWAWNO7RIRCNQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.475332521 (380h45m2.346415566s) [noise=true , address=145.79.227.26:4653 , piece_id=4TEDJLJQK7FLAGKHR2OQYNCCYGNAPO2YQJT2UNFNYZIPXMVV6A7A, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.489574809 (380h45m2.360657855s) [noise=true , address=145.79.227.37:4565 , piece_id=RUIB67LFS5PSVAQQKL7XMI5MBXEVA42X52LTXCPPSRPCGUYWCDGQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.495733106 (380h45m2.366816151s) [noise=true , address=145.79.227.49:4337 , piece_id=6COIQ3HVSY7QT3UGW7FSLAWRKELQZKVAEWCTBAPH37JYFWWADYUQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.540052607 (380h45m2.411135652s) [noise=true , address=145.79.227.51:4410 , piece_id=3HU54QLVMYN7LSYLNKNAVTFYUOM6LIGOJ2A6S5EYMD7DBLHMCWRQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.545014355 (380h45m2.4160974s) [noise=true , address=145.79.227.10:4403 , piece_id=ADNUV4N3MEUYPYJBMXFURSR4JNL6F5HYMS4HFUQCGMWR2N2252TA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.554598926 (380h45m2.425681971s) [noise=true , address=145.79.227.39:4014 , piece_id=RYKJRPSGBJED4J25YP7JS4R42HKA474563X6ZKUBCP7J5MGAWTPQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.560517024 (380h45m2.431600069s) [noise=true , address=145.79.227.38:4977 , piece_id=IUDZFALFSX5QE26GVWMXNP6GP35W3C24HNIHADQPDL6OK2GEUHGA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.593460894 (380h45m2.464543939s) [noise=true , address=45.91.116.4:4937 , piece_id=6ANWFRY3H7T4ETX5VT2FQNKNDSIY2ICSDYTKIZKX6TVM4AXUXIDA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.596454945 (380h45m2.46753799s) [noise=true , address=45.91.116.7:4012 , piece_id=4ZUTJ5QSPSGYLX7ZO34GU3WF3URE3TOEQPM7Q6ISSZVWZPNBZ4XQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:02.829578497 (380h45m2.700661542s) [noise=true , address=145.79.227.8:4330 , piece_id=NNSDYGUPN3QVNZ63QRLERD3XD4SYETHL44YGA6435HKXFMAH5KLA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:03.167529816 (380h45m3.038612861s) [noise=true , address=145.79.227.41:4087 , piece_id=TN4YDWBFCZFTZD7DG5RES7UB3MXSUWZRML5ME3J4AO2XM7IRF2YA, seg_pos={0 0}, upload=334148]: Done uploading piece. err: <nil> Dec 21 12:14:03.167547389 (380h45m3.038630424s) [seg_pos={0 0}, upload=334148]: Segment reached optimal threshold of 13 pieces. Dec 21 12:14:03.167724857 (380h45m3.038807902s) [noise=true , address=145.79.227.29:4942 , piece_id=EHUOHUW4T37BC4QLP5T37JLV35MDMEWF3WQYDDK4IL7FWDDBWFNQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 21 12:14:03.167736028 (380h45m3.038819073s) [noise=true , address=145.79.227.29:4942 , piece_id=EHUOHUW4T37BC4QLP5T37JLV35MDMEWF3WQYDDK4IL7FWDDBWFNQ, seg_pos={0 0}, upload=334148]: Long tail cancellation due to optimal threshold Dec 21 12:14:03.167922023 (380h45m3.039005058s) [noise=true , address=145.79.227.14:4840 , piece_id=MZ3ECARQPDOYZPZZVMJWQFYX4XLAVCYAE62FPFBT4JT6HHF2W5PQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 21 12:14:03.167938504 (380h45m3.039021549s) [noise=true , address=145.79.227.14:4840 , piece_id=MZ3ECARQPDOYZPZZVMJWQFYX4XLAVCYAE62FPFBT4JT6HHF2W5PQ, seg_pos={0 0}, upload=334148]: Long tail cancellation due to optimal threshold Dec 21 12:14:03.168068993 (380h45m3.039152028s) [noise=true , address=145.79.227.36:4519 , piece_id=YVLIPA2GWTENE37FIHWRD4XOF3MBNXJUNLHLG6GBEAN56E5SZOXQ, seg_pos={0 0}, upload=334148]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 21 12:14:03.168076807 (380h45m3.039159852s) [noise=true , address=145.79.227.36:4519 , piece_id=YVLIPA2GWTENE37FIHWRD4XOF3MBNXJUNLHLG6GBEAN56E5SZOXQ, seg_pos={0 0}, upload=334148]: Long tail cancellation due to optimal threshold Dec 21 12:14:03.168096385 (380h45m3.03917943s) [seg_pos={0 0}, upload=334148]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:14:03.168166759 (380h45m3.039249804s) [upload=334148]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:14:03.168172179 (380h45m3.039255224s) [upload=334148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:14:03.460259967 (380h45m3.331343022s) [upload=334148]: Done uploading segments... Dec 21 12:14:03.460287389 (380h45m3.331370434s) [upload=334148]: Upload finished. err: <nil> Duration: 30.002948809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X........................................XX.X..X.X..X.XX] duration: 30.003415937s trace: https://jaeger.dev.storj.io/trace/83063d53317bf20 FAIL (2025-12-21 12:13:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 12:13:13.079029422 (380h44m12.950112457s) [upload=334135]: Starting upload Dec 21 12:13:13.079125104 (380h44m12.950208149s) [upload=334135]: Done starting upload Dec 21 12:13:13.079174688 (380h44m12.950257733s) [upload=334135]: Uploading segments... Dec 21 12:13:13.079441427 (380h44m12.950524462s) [upload=334135]: Got next segment. Inline: false Dec 21 12:13:13.079498424 (380h44m12.95058146s) [upload=334135]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:13:13.079507002 (380h44m12.950590037s) [upload=334135]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:13:13.325474583 (380h44m13.196557618s) [seg_pos={0 0}, upload=334135]: Begin upload segment... Dec 21 12:13:13.325523977 (380h44m13.196607012s) [seg_pos={0 0}, upload=334135]: Done begin upload segment. Dec 21 12:13:13.325561098 (380h44m13.196644133s) [noise=true , address=45.91.116.12:4225 , piece_id=RVDYF4UQJZYFF7NE6UPK7RJ25WRK4VNWJWREZUXIWW6YIQ4VBG2Q, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.325569935 (380h44m13.19665297s) [noise=true , address=145.79.227.27:4690 , piece_id=SK557H5CY5OPDQBWQR5K36IOMP2FR5VNI5AMR3VHPYT7JUUIONPA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.325916004 (380h44m13.196999049s) [noise=true , address=145.79.227.49:4337 , piece_id=A33RZQUPAEPF6UMVLYJ662XCPNC4MXKWIJWH2G3VJEX7JSVF7DIQ, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.326206076 (380h44m13.197289121s) [noise=true , address=45.91.116.13:4155 , piece_id=YH3MIMKP2CEAYTA3ZHEIKYCFGIRPXFQCY6FOXTMARY4RZXXOQPRQ, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.326463416 (380h44m13.197546451s) [noise=true , address=45.91.116.14:4189 , piece_id=QGZKKS7RTYKA6FHBWSRSB6I7QTONKBH2AEZNEKQTYF4B3NCKVFIA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.326736326 (380h44m13.197819361s) [noise=true , address=145.79.227.52:4446 , piece_id=B3WNRULHRA7YWYHKI5BZS4HKF7LGEC2UZYHJWXTE4BNFBGBTGDTQ, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.327128482 (380h44m13.198211517s) [noise=true , address=45.91.116.9:4262 , piece_id=Q7JHBMFPDVV6R3QQTTXJQ5C2PBHBDTGVEADY5OMZQY34T73MPDUA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.327453321 (380h44m13.198536357s) [noise=true , address=145.79.227.45:4231 , piece_id=BLVYMXP6HKJXXKK2ZAQYXMOZHZIJIKNBX5A5OSEVF2ELBX6SFV2A, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.327903769 (380h44m13.198986804s) [noise=true , address=145.79.227.42:4122 , piece_id=SCI326R42BIEQNZVQCULQ27LUIK3QUNBCTXITLECT6F4FLYKHCFA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.328388653 (380h44m13.199471688s) [noise=true , address=145.79.227.47:4266 , piece_id=EUCZ2VG4QZVFMUYAYTGH3KUANGNHPY6T7MYBCJ4SXHLX5YTIB7IA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.328709844 (380h44m13.199792879s) [noise=true , address=145.79.227.2:4115 , piece_id=C6T2Y34Q2YC4ZBQQL4GRRQQGYKFAGAPMBQE5CQWJTWTFS5I4PJOA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.329040122 (380h44m13.200123167s) [noise=true , address=145.79.227.38:4979 , piece_id=6GREWJI4RXBRSX4JHQXLGFR7PEVDRO2BV6JM6EJ5VMFKP7EWGKEQ, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.329334674 (380h44m13.200417709s) [noise=true , address=145.79.227.20:4437 , piece_id=JO7WWZQIW7MP72F4RX6HTAVR2NFPMANFQNNNNJZLPWJYEUSVV5DA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.329690060 (380h44m13.200773095s) [noise=true , address=145.79.227.37:4567 , piece_id=6IUKGCNHXGYVC7QUEDJKOJIPOE6YWPP3AJF2ZH3SPIVKO3RQCOBQ, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.330037041 (380h44m13.201120086s) [noise=true , address=45.91.116.5:4297 , piece_id=NY6HVW4NBK6VFFM5ZZIZLEDZZYLTERQTIYFNXYLF7XDMXG2EBF5A, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:13.330932096 (380h44m13.202015141s) [noise=true , address=45.91.116.6:4975 , piece_id=FUK2Y75K7T3UDENPNQBFUU3ROUQBNY5ACSDLZKLW7R2QYIQ4MBLA, seg_pos={0 0}, upload=334135]: Uploading piece... Dec 21 12:13:15.119068194 (380h44m14.990151239s) [upload=334135]: Next returned nil segment Dec 21 12:13:15.119085979 (380h44m14.990169024s) [upload=334135]: Waiting for error group managing segments... Dec 21 12:13:15.511608414 (380h44m15.382691449s) [noise=true , address=145.79.227.20:4437 , piece_id=JO7WWZQIW7MP72F4RX6HTAVR2NFPMANFQNNNNJZLPWJYEUSVV5DA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.553096905 (380h44m15.42417995s) [noise=true , address=45.91.116.14:4189 , piece_id=QGZKKS7RTYKA6FHBWSRSB6I7QTONKBH2AEZNEKQTYF4B3NCKVFIA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.555352069 (380h44m15.426435104s) [noise=true , address=145.79.227.27:4690 , piece_id=SK557H5CY5OPDQBWQR5K36IOMP2FR5VNI5AMR3VHPYT7JUUIONPA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.587167661 (380h44m15.458250706s) [noise=true , address=145.79.227.38:4979 , piece_id=6GREWJI4RXBRSX4JHQXLGFR7PEVDRO2BV6JM6EJ5VMFKP7EWGKEQ, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.623859047 (380h44m15.494942092s) [noise=true , address=45.91.116.5:4297 , piece_id=NY6HVW4NBK6VFFM5ZZIZLEDZZYLTERQTIYFNXYLF7XDMXG2EBF5A, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.628335952 (380h44m15.499418997s) [noise=true , address=45.91.116.9:4262 , piece_id=Q7JHBMFPDVV6R3QQTTXJQ5C2PBHBDTGVEADY5OMZQY34T73MPDUA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.693769919 (380h44m15.564852964s) [noise=true , address=45.91.116.13:4155 , piece_id=YH3MIMKP2CEAYTA3ZHEIKYCFGIRPXFQCY6FOXTMARY4RZXXOQPRQ, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.957385386 (380h44m15.828468431s) [noise=true , address=45.91.116.6:4975 , piece_id=FUK2Y75K7T3UDENPNQBFUU3ROUQBNY5ACSDLZKLW7R2QYIQ4MBLA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:15.989543359 (380h44m15.860626404s) [noise=true , address=45.91.116.12:4225 , piece_id=RVDYF4UQJZYFF7NE6UPK7RJ25WRK4VNWJWREZUXIWW6YIQ4VBG2Q, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:16.204821945 (380h44m16.07590498s) [noise=true , address=145.79.227.37:4567 , piece_id=6IUKGCNHXGYVC7QUEDJKOJIPOE6YWPP3AJF2ZH3SPIVKO3RQCOBQ, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:16.252256347 (380h44m16.123339382s) [noise=true , address=145.79.227.42:4122 , piece_id=SCI326R42BIEQNZVQCULQ27LUIK3QUNBCTXITLECT6F4FLYKHCFA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:16.269654977 (380h44m16.140738013s) [noise=true , address=145.79.227.2:4115 , piece_id=C6T2Y34Q2YC4ZBQQL4GRRQQGYKFAGAPMBQE5CQWJTWTFS5I4PJOA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:16.277331054 (380h44m16.148414099s) [noise=true , address=145.79.227.45:4231 , piece_id=BLVYMXP6HKJXXKK2ZAQYXMOZHZIJIKNBX5A5OSEVF2ELBX6SFV2A, seg_pos={0 0}, upload=334135]: Done uploading piece. err: <nil> Dec 21 12:13:16.277354268 (380h44m16.148437313s) [seg_pos={0 0}, upload=334135]: Segment reached optimal threshold of 13 pieces. Dec 21 12:13:16.277761784 (380h44m16.148844869s) [noise=true , address=145.79.227.47:4266 , piece_id=EUCZ2VG4QZVFMUYAYTGH3KUANGNHPY6T7MYBCJ4SXHLX5YTIB7IA, seg_pos={0 0}, upload=334135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW): protocol: expected piece hash; context canceled; EOF Dec 21 12:13:16.277773526 (380h44m16.148856571s) [noise=true , address=145.79.227.47:4266 , piece_id=EUCZ2VG4QZVFMUYAYTGH3KUANGNHPY6T7MYBCJ4SXHLX5YTIB7IA, seg_pos={0 0}, upload=334135]: Long tail cancellation due to optimal threshold Dec 21 12:13:16.278338131 (380h44m16.149421176s) [noise=true , address=145.79.227.49:4337 , piece_id=A33RZQUPAEPF6UMVLYJ662XCPNC4MXKWIJWH2G3VJEX7JSVF7DIQ, seg_pos={0 0}, upload=334135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 21 12:13:16.278348571 (380h44m16.149431606s) [noise=true , address=145.79.227.49:4337 , piece_id=A33RZQUPAEPF6UMVLYJ662XCPNC4MXKWIJWH2G3VJEX7JSVF7DIQ, seg_pos={0 0}, upload=334135]: Long tail cancellation due to optimal threshold Dec 21 12:13:16.278678069 (380h44m16.149761124s) [noise=true , address=145.79.227.52:4446 , piece_id=B3WNRULHRA7YWYHKI5BZS4HKF7LGEC2UZYHJWXTE4BNFBGBTGDTQ, seg_pos={0 0}, upload=334135]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: expected piece hash; context canceled; EOF Dec 21 12:13:16.278710771 (380h44m16.149793816s) [noise=true , address=145.79.227.52:4446 , piece_id=B3WNRULHRA7YWYHKI5BZS4HKF7LGEC2UZYHJWXTE4BNFBGBTGDTQ, seg_pos={0 0}, upload=334135]: Long tail cancellation due to optimal threshold Dec 21 12:13:16.278736440 (380h44m16.149819485s) [seg_pos={0 0}, upload=334135]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:13:16.278927173 (380h44m16.150010218s) [upload=334135]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:13:16.278934887 (380h44m16.150017932s) [upload=334135]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:13:16.572167360 (380h44m16.443250395s) [upload=334135]: Done uploading segments... Dec 21 12:13:16.572208498 (380h44m16.443291543s) [upload=334135]: Upload finished. err: <nil> Duration: 30.003438651s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)