Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.020490221s trace: https://jaeger.dev.storj.io/trace/42352d377c8768c FAIL (2025-12-31 10:30:57 UTC) orbiter warning: context deadline exceeded Duration: 6.020505861s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001200366s trace: https://jaeger.dev.storj.io/trace/42352d377c8768c FAIL (2025-12-31 10:30:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001223459s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XXX................XX......X....X.X...X..........X...X.XX.X] duration: 5.00140427s trace: https://jaeger.dev.storj.io/trace/336e2cbac9f60469 FAIL (2025-12-31 10:30:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00142993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.X.....] duration: 6.097168988s trace: https://jaeger.dev.storj.io/trace/624255d1bd63e7c0 FAIL (2025-12-31 10:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ) context deadline exceeded ======LOGS====== 1 10:26:18.740554558 (618h57m18.611637603s) [upload=542405]: Starting upload Dec 31 10:26:18.740632356 (618h57m18.611715401s) [upload=542405]: Done starting upload Dec 31 10:26:18.740825664 (618h57m18.611908699s) [upload=542405]: Uploading segments... Dec 31 10:26:18.740888844 (618h57m18.611971889s) [upload=542405]: Got next segment. Inline: false Dec 31 10:26:18.740916657 (618h57m18.611999693s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:26:18.740920775 (618h57m18.61200381s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:26:18.986262862 (618h57m18.857345907s) [seg_pos={0 0}, upload=542405]: Begin upload segment... Dec 31 10:26:18.986311335 (618h57m18.85739438s) [seg_pos={0 0}, upload=542405]: Done begin upload segment. Dec 31 10:26:18.986339429 (618h57m18.857422464s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986380968 (618h57m18.857464003s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986781189 (618h57m18.857864224s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987120015 (618h57m18.85820323s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987740256 (618h57m18.858823291s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987835317 (618h57m18.858918352s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988061347 (618h57m18.859144382s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988765598 (618h57m18.859848643s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988869666 (618h57m18.859952701s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989556284 (618h57m18.860639329s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989624133 (618h57m18.860707178s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990416001 (618h57m18.861499036s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990676286 (618h57m18.861759331s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991317007 (618h57m18.862400042s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991458727 (618h57m18.862541762s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.992460845 (618h57m18.86354388s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:20.596812097 (618h57m20.467895132s) [upload=542405]: Next returned nil segment Dec 31 10:26:20.596827887 (618h57m20.467910922s) [upload=542405]: Waiting for error group managing segments... Dec 31 10:26:20.976752589 (618h57m20.847835634s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980361010 (618h57m20.851444045s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980443918 (618h57m20.851526963s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.981789170 (618h57m20.852872225s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.984187587 (618h57m20.855270622s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.996770897 (618h57m20.867853942s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.011446331 (618h57m20.882529376s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.063944476 (618h57m20.935027511s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.168926096 (618h57m21.040009131s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.261047939 (618h57m21.132130974s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.265497983 (618h57m21.136581018s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.375358383 (618h57m21.246441418s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424928272 (618h57m21.296011307s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424953110 (618h57m21.296036155s) [seg_pos={0 0}, upload=542405]: Segment reached optimal threshold of 13 pieces. Dec 31 10:26:21.425177436 (618h57m21.296260481s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection Dec 31 10:26:21.425184950 (618h57m21.296267986s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425365836 (618h57m21.296448871s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425373049 (618h57m21.296456084s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425400953 (618h57m21.296483998s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425422243 (618h57m21.296505288s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425465044 (618h57m21.296548089s) [seg_pos={0 0}, upload=542405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:26:21.425550126 (618h57m21.296633161s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:26:21.425557310 (618h57m21.296640345s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:26:21.724438704 (618h57m21.595521739s) [upload=542405]: Done uploading segments... Dec 31 10:26:21.724459824 (618h57m21.595542859s) [upload=542405]: Upload finished. err: <nil> Duration: 6.097197753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X........X......X.....................................] duration: 39.945146269s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 39.945169033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XXX................XX......X....X.X...X..........X...X.XX.] duration: 3.754497511s trace: https://jaeger.dev.storj.io/trace/252239bcbf28e870 FAIL (2025-12-31 10:29:52 UTC) orbiter warning: context deadline exceeded Duration: 3.754527359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.X....] duration: 6.089681518s trace: https://jaeger.dev.storj.io/trace/624255d1bd63e7c0 FAIL (2025-12-31 10:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ) context deadline exceeded ======LOGS====== 1 10:26:18.740554558 (618h57m18.611637603s) [upload=542405]: Starting upload Dec 31 10:26:18.740632356 (618h57m18.611715401s) [upload=542405]: Done starting upload Dec 31 10:26:18.740825664 (618h57m18.611908699s) [upload=542405]: Uploading segments... Dec 31 10:26:18.740888844 (618h57m18.611971889s) [upload=542405]: Got next segment. Inline: false Dec 31 10:26:18.740916657 (618h57m18.611999693s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:26:18.740920775 (618h57m18.61200381s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:26:18.986262862 (618h57m18.857345907s) [seg_pos={0 0}, upload=542405]: Begin upload segment... Dec 31 10:26:18.986311335 (618h57m18.85739438s) [seg_pos={0 0}, upload=542405]: Done begin upload segment. Dec 31 10:26:18.986339429 (618h57m18.857422464s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986380968 (618h57m18.857464003s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986781189 (618h57m18.857864224s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987120015 (618h57m18.85820323s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987740256 (618h57m18.858823291s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987835317 (618h57m18.858918352s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988061347 (618h57m18.859144382s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988765598 (618h57m18.859848643s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988869666 (618h57m18.859952701s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989556284 (618h57m18.860639329s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989624133 (618h57m18.860707178s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990416001 (618h57m18.861499036s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990676286 (618h57m18.861759331s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991317007 (618h57m18.862400042s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991458727 (618h57m18.862541762s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.992460845 (618h57m18.86354388s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:20.596812097 (618h57m20.467895132s) [upload=542405]: Next returned nil segment Dec 31 10:26:20.596827887 (618h57m20.467910922s) [upload=542405]: Waiting for error group managing segments... Dec 31 10:26:20.976752589 (618h57m20.847835634s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980361010 (618h57m20.851444045s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980443918 (618h57m20.851526963s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.981789170 (618h57m20.852872225s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.984187587 (618h57m20.855270622s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.996770897 (618h57m20.867853942s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.011446331 (618h57m20.882529376s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.063944476 (618h57m20.935027511s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.168926096 (618h57m21.040009131s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.261047939 (618h57m21.132130974s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.265497983 (618h57m21.136581018s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.375358383 (618h57m21.246441418s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424928272 (618h57m21.296011307s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424953110 (618h57m21.296036155s) [seg_pos={0 0}, upload=542405]: Segment reached optimal threshold of 13 pieces. Dec 31 10:26:21.425177436 (618h57m21.296260481s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection Dec 31 10:26:21.425184950 (618h57m21.296267986s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425365836 (618h57m21.296448871s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425373049 (618h57m21.296456084s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425400953 (618h57m21.296483998s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425422243 (618h57m21.296505288s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425465044 (618h57m21.296548089s) [seg_pos={0 0}, upload=542405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:26:21.425550126 (618h57m21.296633161s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:26:21.425557310 (618h57m21.296640345s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:26:21.724438704 (618h57m21.595521739s) [upload=542405]: Done uploading segments... Dec 31 10:26:21.724459824 (618h57m21.595542859s) [upload=542405]: Upload finished. err: <nil> Duration: 6.089711164s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X........X......X....................................] duration: 35.229218921s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:29:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 35.229232318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.X...] duration: 25.517055389s trace: https://jaeger.dev.storj.io/trace/624255d1bd63e7c0 FAIL (2025-12-31 10:29:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ) context deadline exceeded ======LOGS====== 1 10:26:18.740554558 (618h57m18.611637603s) [upload=542405]: Starting upload Dec 31 10:26:18.740632356 (618h57m18.611715401s) [upload=542405]: Done starting upload Dec 31 10:26:18.740825664 (618h57m18.611908699s) [upload=542405]: Uploading segments... Dec 31 10:26:18.740888844 (618h57m18.611971889s) [upload=542405]: Got next segment. Inline: false Dec 31 10:26:18.740916657 (618h57m18.611999693s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:26:18.740920775 (618h57m18.61200381s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:26:18.986262862 (618h57m18.857345907s) [seg_pos={0 0}, upload=542405]: Begin upload segment... Dec 31 10:26:18.986311335 (618h57m18.85739438s) [seg_pos={0 0}, upload=542405]: Done begin upload segment. Dec 31 10:26:18.986339429 (618h57m18.857422464s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986380968 (618h57m18.857464003s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986781189 (618h57m18.857864224s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987120015 (618h57m18.85820323s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987740256 (618h57m18.858823291s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987835317 (618h57m18.858918352s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988061347 (618h57m18.859144382s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988765598 (618h57m18.859848643s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988869666 (618h57m18.859952701s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989556284 (618h57m18.860639329s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989624133 (618h57m18.860707178s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990416001 (618h57m18.861499036s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990676286 (618h57m18.861759331s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991317007 (618h57m18.862400042s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991458727 (618h57m18.862541762s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.992460845 (618h57m18.86354388s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:20.596812097 (618h57m20.467895132s) [upload=542405]: Next returned nil segment Dec 31 10:26:20.596827887 (618h57m20.467910922s) [upload=542405]: Waiting for error group managing segments... Dec 31 10:26:20.976752589 (618h57m20.847835634s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980361010 (618h57m20.851444045s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980443918 (618h57m20.851526963s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.981789170 (618h57m20.852872225s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.984187587 (618h57m20.855270622s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.996770897 (618h57m20.867853942s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.011446331 (618h57m20.882529376s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.063944476 (618h57m20.935027511s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.168926096 (618h57m21.040009131s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.261047939 (618h57m21.132130974s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.265497983 (618h57m21.136581018s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.375358383 (618h57m21.246441418s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424928272 (618h57m21.296011307s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424953110 (618h57m21.296036155s) [seg_pos={0 0}, upload=542405]: Segment reached optimal threshold of 13 pieces. Dec 31 10:26:21.425177436 (618h57m21.296260481s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection Dec 31 10:26:21.425184950 (618h57m21.296267986s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425365836 (618h57m21.296448871s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425373049 (618h57m21.296456084s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425400953 (618h57m21.296483998s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425422243 (618h57m21.296505288s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425465044 (618h57m21.296548089s) [seg_pos={0 0}, upload=542405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:26:21.425550126 (618h57m21.296633161s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:26:21.425557310 (618h57m21.296640345s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:26:21.724438704 (618h57m21.595521739s) [upload=542405]: Done uploading segments... Dec 31 10:26:21.724459824 (618h57m21.595542859s) [upload=542405]: Upload finished. err: <nil> Duration: 25.517083692s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XXX................XX......X....X.X...X..........X...X.XX] duration: 5.001389313s trace: https://jaeger.dev.storj.io/trace/252239bcbf28e870 FAIL (2025-12-31 10:28:44 UTC) orbiter warning: context deadline exceeded Duration: 5.00140307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X........X......X...................................] duration: 33.168132856s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 33.168158323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....XXX................XX......X....X.X...X..........X...X.X] duration: 5.000838233s trace: https://jaeger.dev.storj.io/trace/6ba4dea3f88a056b FAIL (2025-12-31 10:28:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000859434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.X..] duration: 5.718497871s trace: https://jaeger.dev.storj.io/trace/624255d1bd63e7c0 FAIL (2025-12-31 10:27:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ) context deadline exceeded ======LOGS====== 1 10:26:18.740554558 (618h57m18.611637603s) [upload=542405]: Starting upload Dec 31 10:26:18.740632356 (618h57m18.611715401s) [upload=542405]: Done starting upload Dec 31 10:26:18.740825664 (618h57m18.611908699s) [upload=542405]: Uploading segments... Dec 31 10:26:18.740888844 (618h57m18.611971889s) [upload=542405]: Got next segment. Inline: false Dec 31 10:26:18.740916657 (618h57m18.611999693s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:26:18.740920775 (618h57m18.61200381s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:26:18.986262862 (618h57m18.857345907s) [seg_pos={0 0}, upload=542405]: Begin upload segment... Dec 31 10:26:18.986311335 (618h57m18.85739438s) [seg_pos={0 0}, upload=542405]: Done begin upload segment. Dec 31 10:26:18.986339429 (618h57m18.857422464s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986380968 (618h57m18.857464003s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986781189 (618h57m18.857864224s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987120015 (618h57m18.85820323s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987740256 (618h57m18.858823291s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987835317 (618h57m18.858918352s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988061347 (618h57m18.859144382s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988765598 (618h57m18.859848643s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988869666 (618h57m18.859952701s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989556284 (618h57m18.860639329s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989624133 (618h57m18.860707178s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990416001 (618h57m18.861499036s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990676286 (618h57m18.861759331s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991317007 (618h57m18.862400042s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991458727 (618h57m18.862541762s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.992460845 (618h57m18.86354388s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:20.596812097 (618h57m20.467895132s) [upload=542405]: Next returned nil segment Dec 31 10:26:20.596827887 (618h57m20.467910922s) [upload=542405]: Waiting for error group managing segments... Dec 31 10:26:20.976752589 (618h57m20.847835634s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980361010 (618h57m20.851444045s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980443918 (618h57m20.851526963s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.981789170 (618h57m20.852872225s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.984187587 (618h57m20.855270622s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.996770897 (618h57m20.867853942s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.011446331 (618h57m20.882529376s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.063944476 (618h57m20.935027511s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.168926096 (618h57m21.040009131s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.261047939 (618h57m21.132130974s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.265497983 (618h57m21.136581018s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.375358383 (618h57m21.246441418s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424928272 (618h57m21.296011307s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424953110 (618h57m21.296036155s) [seg_pos={0 0}, upload=542405]: Segment reached optimal threshold of 13 pieces. Dec 31 10:26:21.425177436 (618h57m21.296260481s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection Dec 31 10:26:21.425184950 (618h57m21.296267986s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425365836 (618h57m21.296448871s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425373049 (618h57m21.296456084s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425400953 (618h57m21.296483998s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425422243 (618h57m21.296505288s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425465044 (618h57m21.296548089s) [seg_pos={0 0}, upload=542405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:26:21.425550126 (618h57m21.296633161s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:26:21.425557310 (618h57m21.296640345s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:26:21.724438704 (618h57m21.595521739s) [upload=542405]: Done uploading segments... Dec 31 10:26:21.724459824 (618h57m21.595542859s) [upload=542405]: Upload finished. err: <nil> Duration: 5.718525505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X........X......X..................................] duration: 29.999825205s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 29.999851607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.....XXX................XX......X....X.X...X..........X...X.] duration: 3.549185531s trace: https://jaeger.dev.storj.io/trace/f0642a4da137d2e FAIL (2025-12-31 10:27:24 UTC) orbiter warning: context deadline exceeded Duration: 3.549213866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.X.] duration: 5.965754198s trace: https://jaeger.dev.storj.io/trace/624255d1bd63e7c0 FAIL (2025-12-31 10:26:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ) context deadline exceeded ======LOGS====== 1 10:26:18.740554558 (618h57m18.611637603s) [upload=542405]: Starting upload Dec 31 10:26:18.740632356 (618h57m18.611715401s) [upload=542405]: Done starting upload Dec 31 10:26:18.740825664 (618h57m18.611908699s) [upload=542405]: Uploading segments... Dec 31 10:26:18.740888844 (618h57m18.611971889s) [upload=542405]: Got next segment. Inline: false Dec 31 10:26:18.740916657 (618h57m18.611999693s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:26:18.740920775 (618h57m18.61200381s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:26:18.986262862 (618h57m18.857345907s) [seg_pos={0 0}, upload=542405]: Begin upload segment... Dec 31 10:26:18.986311335 (618h57m18.85739438s) [seg_pos={0 0}, upload=542405]: Done begin upload segment. Dec 31 10:26:18.986339429 (618h57m18.857422464s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986380968 (618h57m18.857464003s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986781189 (618h57m18.857864224s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987120015 (618h57m18.85820323s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987740256 (618h57m18.858823291s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987835317 (618h57m18.858918352s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988061347 (618h57m18.859144382s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988765598 (618h57m18.859848643s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988869666 (618h57m18.859952701s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989556284 (618h57m18.860639329s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989624133 (618h57m18.860707178s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990416001 (618h57m18.861499036s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990676286 (618h57m18.861759331s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991317007 (618h57m18.862400042s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991458727 (618h57m18.862541762s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.992460845 (618h57m18.86354388s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:20.596812097 (618h57m20.467895132s) [upload=542405]: Next returned nil segment Dec 31 10:26:20.596827887 (618h57m20.467910922s) [upload=542405]: Waiting for error group managing segments... Dec 31 10:26:20.976752589 (618h57m20.847835634s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980361010 (618h57m20.851444045s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980443918 (618h57m20.851526963s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.981789170 (618h57m20.852872225s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.984187587 (618h57m20.855270622s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.996770897 (618h57m20.867853942s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.011446331 (618h57m20.882529376s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.063944476 (618h57m20.935027511s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.168926096 (618h57m21.040009131s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.261047939 (618h57m21.132130974s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.265497983 (618h57m21.136581018s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.375358383 (618h57m21.246441418s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424928272 (618h57m21.296011307s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424953110 (618h57m21.296036155s) [seg_pos={0 0}, upload=542405]: Segment reached optimal threshold of 13 pieces. Dec 31 10:26:21.425177436 (618h57m21.296260481s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection Dec 31 10:26:21.425184950 (618h57m21.296267986s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425365836 (618h57m21.296448871s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425373049 (618h57m21.296456084s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425400953 (618h57m21.296483998s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425422243 (618h57m21.296505288s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425465044 (618h57m21.296548089s) [seg_pos={0 0}, upload=542405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:26:21.425550126 (618h57m21.296633161s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:26:21.425557310 (618h57m21.296640345s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:26:21.724438704 (618h57m21.595521739s) [upload=542405]: Done uploading segments... Dec 31 10:26:21.724459824 (618h57m21.595542859s) [upload=542405]: Upload finished. err: <nil> Duration: 5.965771362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.X] duration: 30.002748288s trace: https://jaeger.dev.storj.io/trace/624255d1bd63e7c0 FAIL (2025-12-31 10:26:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: 4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ) context deadline exceeded ======LOGS====== 1 10:26:18.740554558 (618h57m18.611637603s) [upload=542405]: Starting upload Dec 31 10:26:18.740632356 (618h57m18.611715401s) [upload=542405]: Done starting upload Dec 31 10:26:18.740825664 (618h57m18.611908699s) [upload=542405]: Uploading segments... Dec 31 10:26:18.740888844 (618h57m18.611971889s) [upload=542405]: Got next segment. Inline: false Dec 31 10:26:18.740916657 (618h57m18.611999693s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:26:18.740920775 (618h57m18.61200381s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:26:18.986262862 (618h57m18.857345907s) [seg_pos={0 0}, upload=542405]: Begin upload segment... Dec 31 10:26:18.986311335 (618h57m18.85739438s) [seg_pos={0 0}, upload=542405]: Done begin upload segment. Dec 31 10:26:18.986339429 (618h57m18.857422464s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986380968 (618h57m18.857464003s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.986781189 (618h57m18.857864224s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987120015 (618h57m18.85820323s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987740256 (618h57m18.858823291s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.987835317 (618h57m18.858918352s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988061347 (618h57m18.859144382s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988765598 (618h57m18.859848643s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.988869666 (618h57m18.859952701s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989556284 (618h57m18.860639329s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.989624133 (618h57m18.860707178s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990416001 (618h57m18.861499036s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.990676286 (618h57m18.861759331s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991317007 (618h57m18.862400042s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.991458727 (618h57m18.862541762s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:18.992460845 (618h57m18.86354388s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Uploading piece... Dec 31 10:26:20.596812097 (618h57m20.467895132s) [upload=542405]: Next returned nil segment Dec 31 10:26:20.596827887 (618h57m20.467910922s) [upload=542405]: Waiting for error group managing segments... Dec 31 10:26:20.976752589 (618h57m20.847835634s) [noise=true , address=145.79.227.46:4603 , piece_id=BZHIGSBQ4J2I3TD3GXL3QD4ET3QLXWU6HQNMMGOCJ37PA262TWWA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980361010 (618h57m20.851444045s) [noise=true , address=145.79.227.42:4121 , piece_id=GB4YG7FHY4QRHSHZHRFPNCQAWVRW66BYUEHM7MGGMSEJWWXZNLDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.980443918 (618h57m20.851526963s) [noise=true , address=145.79.227.22:4511 , piece_id=GO7BDZ22ARWAJ6TGDYP22XFKWIPDHJPUA5PE4UQVNOJB2H44Z7NA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.981789170 (618h57m20.852872225s) [noise=true , address=145.79.227.31:4797 , piece_id=2AADONZ7WIK6C77RFJKX76IODC3N7FLB4D3L2ZGRYBQL4EQWMQTA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.984187587 (618h57m20.855270622s) [noise=true , address=145.79.227.45:4229 , piece_id=KOXZ5P7O4KKHPSZFGBZ2EMEMJJBIE64X47RSUCL65L5POTXRHRQA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:20.996770897 (618h57m20.867853942s) [noise=true , address=145.79.227.51:4410 , piece_id=JRRQPPXYGYGG6BCRQEBEW2UJFUC7KI4R5FI43KHEROUMOHIW7FEQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.011446331 (618h57m20.882529376s) [noise=true , address=145.79.227.36:4519 , piece_id=SUB7XKYGLQ4324CN6DZMH7FYH55HPLHSZNIOS2QG4YNF3FP72FHQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.063944476 (618h57m20.935027511s) [noise=true , address=145.79.227.4:4185 , piece_id=SCFLIQDFBZU63GSWGIKT47CFUNVOYUWYMQICLCX7XU4SPHR2AXUQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.168926096 (618h57m21.040009131s) [noise=true , address=77.90.31.16:4735 , piece_id=KKWRWHHWUTHH5ZHPEYQBYLY3RINRRI4527XXIML2BCURGO2DIVNQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.261047939 (618h57m21.132130974s) [noise=true , address=77.90.31.18:4808 , piece_id=XG34SZGPC4IRHVLZGIFHGULG35HL3T6MJ2ZXXY7M5BVNPBM2TMDA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.265497983 (618h57m21.136581018s) [noise=true , address=77.90.31.10:4518 , piece_id=4ZN52TOXV4O2RTDHFYUG6JJ5UAHMTQEPHJT2OFV36WLCZ6QPEDPQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.375358383 (618h57m21.246441418s) [noise=true , address=45.91.116.10:4082 , piece_id=BXIOM43G6T6SBWAOEZRDP7MRR5CSBGMLDGECBBI3HQHN65BO32BQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424928272 (618h57m21.296011307s) [noise=true , address=45.91.116.8:4047 , piece_id=GLA3TXF2N7Q35DQTI6MJG47KGHFSHFJSI73HR23YSE2QWZMR6TSA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: <nil> Dec 31 10:26:21.424953110 (618h57m21.296036155s) [seg_pos={0 0}, upload=542405]: Segment reached optimal threshold of 13 pieces. Dec 31 10:26:21.425177436 (618h57m21.296260481s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38698->45.91.116.6:4975: use of closed network connection Dec 31 10:26:21.425184950 (618h57m21.296267986s) [noise=true , address=45.91.116.6:4975 , piece_id=QRZK2CO3SIS6APBPQOXP2DADCWXYVNGSE43XD5JPOVKF4EU3QDDQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425365836 (618h57m21.296448871s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425373049 (618h57m21.296456084s) [noise=true , address=77.90.31.14:4664 , piece_id=OFOUCJTJ3ZRC3NH5YBJB6FRS4UFZMZUT7QYEOPGODRXTBC2MVLBA, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425400953 (618h57m21.296483998s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 31 10:26:21.425422243 (618h57m21.296505288s) [noise=true , address=45.91.116.9:4262 , piece_id=3MD6G4MUOD4JRZWZIYY7BXRZ52IBPYFMIMBS5UEUVZO62ZMI7YGQ, seg_pos={0 0}, upload=542405]: Long tail cancellation due to optimal threshold Dec 31 10:26:21.425465044 (618h57m21.296548089s) [seg_pos={0 0}, upload=542405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:26:21.425550126 (618h57m21.296633161s) [upload=542405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:26:21.425557310 (618h57m21.296640345s) [upload=542405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:26:21.724438704 (618h57m21.595521739s) [upload=542405]: Done uploading segments... Dec 31 10:26:21.724459824 (618h57m21.595542859s) [upload=542405]: Upload finished. err: <nil> Duration: 30.002784557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X........X......X.................................] duration: 49.906951225s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:26:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 49.90698009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [......XXX................XX......X....X.X...X..........X...X] duration: 5.001677632s trace: https://jaeger.dev.storj.io/trace/f0642a4da137d2e FAIL (2025-12-31 10:26:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001699533s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X.] duration: 19.643680695s trace: https://jaeger.dev.storj.io/trace/76f3a0a363bf601f FAIL (2025-12-31 10:25:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 7GZUZM4E4QNEJCWE2QCUBIO3RLVBGJVESMNMLRH4TB4I5JCWIYYA) context deadline exceeded; piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: MWM3HZG5XAS4QFOD2OBZMDT663HMTFZGWPCIM2XQVUTTI7YJBQ2A) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: 2SUEZOMGEEPCHYD32C7XDLZJIL7ZJMHMT25EZVDVALPYOO5BS76A) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NEN2X6QH2KISVRGSPB3A2H3WEXAQA5OPJO4NGRW5LXGHLBHHLOXQ) context deadline exceeded ======LOGS====== Dec 31 10:24:01.521154945 (618h55m1.39223799s) [upload=542372]: Starting upload Dec 31 10:24:01.521260045 (618h55m1.39234309s) [upload=542372]: Done starting upload Dec 31 10:24:01.521299941 (618h55m1.392382986s) [upload=542372]: Uploading segments... Dec 31 10:24:01.521570145 (618h55m1.392653201s) [upload=542372]: Got next segment. Inline: false Dec 31 10:24:01.521606535 (618h55m1.39268958s) [upload=542372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:24:01.521612125 (618h55m1.39269517s) [upload=542372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:24:01.768421337 (618h55m1.639504402s) [seg_pos={0 0}, upload=542372]: Begin upload segment... Dec 31 10:24:01.768480009 (618h55m1.639563054s) [seg_pos={0 0}, upload=542372]: Done begin upload segment. Dec 31 10:24:01.768515386 (618h55m1.639598431s) [noise=true , address=145.79.227.32:4836 , piece_id=MWM3HZG5XAS4QFOD2OBZMDT663HMTFZGWPCIM2XQVUTTI7YJBQ2A, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.774158341 (618h55m1.645241386s) [noise=true , address=145.79.227.38:4979 , piece_id=2SUEZOMGEEPCHYD32C7XDLZJIL7ZJMHMT25EZVDVALPYOO5BS76A, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.775884649 (618h55m1.646967694s) [noise=true , address=145.79.227.41:4086 , piece_id=VPJMKGMAAGAXL2VFQA27VUWY42S2J4HJXZD7SZ2TLRV6J6U2NW2Q, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.775922140 (618h55m1.647005185s) [noise=true , address=45.91.116.13:4153 , piece_id=KON2KEYYBP3C7SNETAZ3CWHT4PLZWIPDWCZKLROO7VZGOR6B3GJQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.777566731 (618h55m1.648649776s) [noise=true , address=145.79.227.47:4267 , piece_id=OQ6RYTP7ETYWECXVIYZFY24PL7UVSEO6PIQA6PDFUGYATVAS6NCA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.777772433 (618h55m1.648855468s) [noise=true , address=45.91.116.9:4264 , piece_id=WSP4DN4EBC2NM6B6LHPFHC3E7NLULMFO2FCDGUVPN6XU3E766BRA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.779487889 (618h55m1.650570934s) [noise=true , address=145.79.227.14:4837 , piece_id=CF3WPQBUBJ4GVNOPYTGYK6YVUDEW64V77MIZQM6ODUNQR2JKXUTA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.779629018 (618h55m1.650712063s) [noise=true , address=145.79.227.48:4303 , piece_id=O7YGNBM7YSVRRPZSC523YTU22EOKZV37F2LNAQUJFKLOO6YDZUGQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.781024385 (618h55m1.65210742s) [noise=true , address=77.90.31.13:4629 , piece_id=7GZUZM4E4QNEJCWE2QCUBIO3RLVBGJVESMNMLRH4TB4I5JCWIYYA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.782447435 (618h55m1.65353048s) [noise=true , address=145.79.227.45:4229 , piece_id=ZMNMN6EPBVAKHC4WICUKQIREOPVW3DIIFB6Z6PT4NUHFCFPDWTUA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.783892626 (618h55m1.654975671s) [noise=true , address=45.91.116.12:4227 , piece_id=A53DDZDX36E7ERSK5OPKMKNW3KSMZYUB2ARQHZJHEP3XGXVD2NUA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.785150722 (618h55m1.656233757s) [noise=true , address=145.79.227.19:4078 , piece_id=7IYBWUPQV3WHQE7KHVQCTEBMSPS7VAPEZ2CKNBIMIQ2ELPOL5KOA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.786530830 (618h55m1.657613875s) [noise=true , address=145.79.227.20:4438 , piece_id=FR5LTEP2QRIJDWMF3TMHDWJEW7UB6K6GO3BRGSHLPS627HOH4DPA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.789906237 (618h55m1.660989282s) [noise=true , address=77.90.31.10:4517 , piece_id=WGIZERW6GO6IQWSOO72SFVNYT47XEM36RHIOUWFM6EKQ4KXCMBGA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.794157212 (618h55m1.665240257s) [noise=true , address=77.90.31.14:4661 , piece_id=6SZUKP6NQ2D75OMAHF2CKQLE5UW64MROCTFO5S3ZQI4LVH5OKULQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.794372942 (618h55m1.665455987s) [noise=true , address=145.79.227.15:4876 , piece_id=NEN2X6QH2KISVRGSPB3A2H3WEXAQA5OPJO4NGRW5LXGHLBHHLOXQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:03.486182678 (618h55m3.357265723s) [upload=542372]: Next returned nil segment Dec 31 10:24:03.486209840 (618h55m3.357292885s) [upload=542372]: Waiting for error group managing segments... Dec 31 10:24:03.767935888 (618h55m3.639018933s) [noise=true , address=145.79.227.38:4979 , piece_id=2SUEZOMGEEPCHYD32C7XDLZJIL7ZJMHMT25EZVDVALPYOO5BS76A, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.768722305 (618h55m3.63980535s) [noise=true , address=145.79.227.48:4303 , piece_id=O7YGNBM7YSVRRPZSC523YTU22EOKZV37F2LNAQUJFKLOO6YDZUGQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.775448272 (618h55m3.646531317s) [noise=true , address=145.79.227.15:4876 , piece_id=NEN2X6QH2KISVRGSPB3A2H3WEXAQA5OPJO4NGRW5LXGHLBHHLOXQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.787493989 (618h55m3.658577044s) [noise=true , address=145.79.227.41:4086 , piece_id=VPJMKGMAAGAXL2VFQA27VUWY42S2J4HJXZD7SZ2TLRV6J6U2NW2Q, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.788437586 (618h55m3.659520621s) [noise=true , address=145.79.227.32:4836 , piece_id=MWM3HZG5XAS4QFOD2OBZMDT663HMTFZGWPCIM2XQVUTTI7YJBQ2A, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.791913014 (618h55m3.662996059s) [noise=true , address=145.79.227.45:4229 , piece_id=ZMNMN6EPBVAKHC4WICUKQIREOPVW3DIIFB6Z6PT4NUHFCFPDWTUA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.793094814 (618h55m3.664177879s) [noise=true , address=145.79.227.19:4078 , piece_id=7IYBWUPQV3WHQE7KHVQCTEBMSPS7VAPEZ2CKNBIMIQ2ELPOL5KOA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.794014435 (618h55m3.66509749s) [noise=true , address=145.79.227.47:4267 , piece_id=OQ6RYTP7ETYWECXVIYZFY24PL7UVSEO6PIQA6PDFUGYATVAS6NCA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.858627117 (618h55m3.729710163s) [noise=true , address=145.79.227.14:4837 , piece_id=CF3WPQBUBJ4GVNOPYTGYK6YVUDEW64V77MIZQM6ODUNQR2JKXUTA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.878179428 (618h55m3.749262473s) [noise=true , address=145.79.227.20:4438 , piece_id=FR5LTEP2QRIJDWMF3TMHDWJEW7UB6K6GO3BRGSHLPS627HOH4DPA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.198003230 (618h55m4.069086275s) [noise=true , address=45.91.116.9:4264 , piece_id=WSP4DN4EBC2NM6B6LHPFHC3E7NLULMFO2FCDGUVPN6XU3E766BRA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.413620056 (618h55m4.284703101s) [noise=true , address=77.90.31.14:4661 , piece_id=6SZUKP6NQ2D75OMAHF2CKQLE5UW64MROCTFO5S3ZQI4LVH5OKULQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.784331816 (618h55m4.655414861s) [noise=true , address=77.90.31.13:4629 , piece_id=7GZUZM4E4QNEJCWE2QCUBIO3RLVBGJVESMNMLRH4TB4I5JCWIYYA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.784357224 (618h55m4.655440299s) [seg_pos={0 0}, upload=542372]: Segment reached optimal threshold of 13 pieces. Dec 31 10:24:04.784615215 (618h55m4.65569826s) [noise=true , address=45.91.116.13:4153 , piece_id=KON2KEYYBP3C7SNETAZ3CWHT4PLZWIPDWCZKLROO7VZGOR6B3GJQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 10:24:04.784625245 (618h55m4.65570828s) [noise=true , address=45.91.116.13:4153 , piece_id=KON2KEYYBP3C7SNETAZ3CWHT4PLZWIPDWCZKLROO7VZGOR6B3GJQ, seg_pos={0 0}, upload=542372]: Long tail cancellation due to optimal threshold Dec 31 10:24:04.784889367 (618h55m4.655972402s) [noise=true , address=45.91.116.12:4227 , piece_id=A53DDZDX36E7ERSK5OPKMKNW3KSMZYUB2ARQHZJHEP3XGXVD2NUA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 10:24:04.784896462 (618h55m4.655979497s) [noise=true , address=45.91.116.12:4227 , piece_id=A53DDZDX36E7ERSK5OPKMKNW3KSMZYUB2ARQHZJHEP3XGXVD2NUA, seg_pos={0 0}, upload=542372]: Long tail cancellation due to optimal threshold Dec 31 10:24:04.784904366 (618h55m4.655987411s) [noise=true , address=77.90.31.10:4517 , piece_id=WGIZERW6GO6IQWSOO72SFVNYT47XEM36RHIOUWFM6EKQ4KXCMBGA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 31 10:24:04.784936928 (618h55m4.656019973s) [noise=true , address=77.90.31.10:4517 , piece_id=WGIZERW6GO6IQWSOO72SFVNYT47XEM36RHIOUWFM6EKQ4KXCMBGA, seg_pos={0 0}, upload=542372]: Long tail cancellation due to optimal threshold Dec 31 10:24:04.784966214 (618h55m4.656049259s) [seg_pos={0 0}, upload=542372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:24:04.785146297 (618h55m4.656229342s) [upload=542372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:24:04.785154983 (618h55m4.656238028s) [upload=542372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:24:05.089489080 (618h55m4.960572126s) [upload=542372]: Done uploading segments... Dec 31 10:24:05.089515852 (618h55m4.960598897s) [upload=542372]: Upload finished. err: <nil> Duration: 19.643712306s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......XXX................XX......X....X.X...X..........X...] duration: 3.542773832s trace: https://jaeger.dev.storj.io/trace/68a4b215c6c93ce8 FAIL (2025-12-31 10:25:17 UTC) orbiter warning: context deadline exceeded Duration: 3.542802906s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X........X......X................................] duration: 15.312489505s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 15.312520474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........XXX................XX......X....X.X...X..........X..] duration: 3.731088581s trace: https://jaeger.dev.storj.io/trace/68a4b215c6c93ce8 FAIL (2025-12-31 10:24:50 UTC) orbiter warning: context deadline exceeded Duration: 3.731114691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X........X......X...............................] duration: 37.491543229s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:24:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 37.491556214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.X] duration: 30.003755165s trace: https://jaeger.dev.storj.io/trace/76f3a0a363bf601f FAIL (2025-12-31 10:24:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 7GZUZM4E4QNEJCWE2QCUBIO3RLVBGJVESMNMLRH4TB4I5JCWIYYA) context deadline exceeded; piecestore: (Node ID: 1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, Piece ID: MWM3HZG5XAS4QFOD2OBZMDT663HMTFZGWPCIM2XQVUTTI7YJBQ2A) context deadline exceeded; piecestore: (Node ID: 12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, Piece ID: 2SUEZOMGEEPCHYD32C7XDLZJIL7ZJMHMT25EZVDVALPYOO5BS76A) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NEN2X6QH2KISVRGSPB3A2H3WEXAQA5OPJO4NGRW5LXGHLBHHLOXQ) context deadline exceeded ======LOGS====== Dec 31 10:24:01.521154945 (618h55m1.39223799s) [upload=542372]: Starting upload Dec 31 10:24:01.521260045 (618h55m1.39234309s) [upload=542372]: Done starting upload Dec 31 10:24:01.521299941 (618h55m1.392382986s) [upload=542372]: Uploading segments... Dec 31 10:24:01.521570145 (618h55m1.392653201s) [upload=542372]: Got next segment. Inline: false Dec 31 10:24:01.521606535 (618h55m1.39268958s) [upload=542372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:24:01.521612125 (618h55m1.39269517s) [upload=542372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:24:01.768421337 (618h55m1.639504402s) [seg_pos={0 0}, upload=542372]: Begin upload segment... Dec 31 10:24:01.768480009 (618h55m1.639563054s) [seg_pos={0 0}, upload=542372]: Done begin upload segment. Dec 31 10:24:01.768515386 (618h55m1.639598431s) [noise=true , address=145.79.227.32:4836 , piece_id=MWM3HZG5XAS4QFOD2OBZMDT663HMTFZGWPCIM2XQVUTTI7YJBQ2A, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.774158341 (618h55m1.645241386s) [noise=true , address=145.79.227.38:4979 , piece_id=2SUEZOMGEEPCHYD32C7XDLZJIL7ZJMHMT25EZVDVALPYOO5BS76A, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.775884649 (618h55m1.646967694s) [noise=true , address=145.79.227.41:4086 , piece_id=VPJMKGMAAGAXL2VFQA27VUWY42S2J4HJXZD7SZ2TLRV6J6U2NW2Q, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.775922140 (618h55m1.647005185s) [noise=true , address=45.91.116.13:4153 , piece_id=KON2KEYYBP3C7SNETAZ3CWHT4PLZWIPDWCZKLROO7VZGOR6B3GJQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.777566731 (618h55m1.648649776s) [noise=true , address=145.79.227.47:4267 , piece_id=OQ6RYTP7ETYWECXVIYZFY24PL7UVSEO6PIQA6PDFUGYATVAS6NCA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.777772433 (618h55m1.648855468s) [noise=true , address=45.91.116.9:4264 , piece_id=WSP4DN4EBC2NM6B6LHPFHC3E7NLULMFO2FCDGUVPN6XU3E766BRA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.779487889 (618h55m1.650570934s) [noise=true , address=145.79.227.14:4837 , piece_id=CF3WPQBUBJ4GVNOPYTGYK6YVUDEW64V77MIZQM6ODUNQR2JKXUTA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.779629018 (618h55m1.650712063s) [noise=true , address=145.79.227.48:4303 , piece_id=O7YGNBM7YSVRRPZSC523YTU22EOKZV37F2LNAQUJFKLOO6YDZUGQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.781024385 (618h55m1.65210742s) [noise=true , address=77.90.31.13:4629 , piece_id=7GZUZM4E4QNEJCWE2QCUBIO3RLVBGJVESMNMLRH4TB4I5JCWIYYA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.782447435 (618h55m1.65353048s) [noise=true , address=145.79.227.45:4229 , piece_id=ZMNMN6EPBVAKHC4WICUKQIREOPVW3DIIFB6Z6PT4NUHFCFPDWTUA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.783892626 (618h55m1.654975671s) [noise=true , address=45.91.116.12:4227 , piece_id=A53DDZDX36E7ERSK5OPKMKNW3KSMZYUB2ARQHZJHEP3XGXVD2NUA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.785150722 (618h55m1.656233757s) [noise=true , address=145.79.227.19:4078 , piece_id=7IYBWUPQV3WHQE7KHVQCTEBMSPS7VAPEZ2CKNBIMIQ2ELPOL5KOA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.786530830 (618h55m1.657613875s) [noise=true , address=145.79.227.20:4438 , piece_id=FR5LTEP2QRIJDWMF3TMHDWJEW7UB6K6GO3BRGSHLPS627HOH4DPA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.789906237 (618h55m1.660989282s) [noise=true , address=77.90.31.10:4517 , piece_id=WGIZERW6GO6IQWSOO72SFVNYT47XEM36RHIOUWFM6EKQ4KXCMBGA, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.794157212 (618h55m1.665240257s) [noise=true , address=77.90.31.14:4661 , piece_id=6SZUKP6NQ2D75OMAHF2CKQLE5UW64MROCTFO5S3ZQI4LVH5OKULQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:01.794372942 (618h55m1.665455987s) [noise=true , address=145.79.227.15:4876 , piece_id=NEN2X6QH2KISVRGSPB3A2H3WEXAQA5OPJO4NGRW5LXGHLBHHLOXQ, seg_pos={0 0}, upload=542372]: Uploading piece... Dec 31 10:24:03.486182678 (618h55m3.357265723s) [upload=542372]: Next returned nil segment Dec 31 10:24:03.486209840 (618h55m3.357292885s) [upload=542372]: Waiting for error group managing segments... Dec 31 10:24:03.767935888 (618h55m3.639018933s) [noise=true , address=145.79.227.38:4979 , piece_id=2SUEZOMGEEPCHYD32C7XDLZJIL7ZJMHMT25EZVDVALPYOO5BS76A, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.768722305 (618h55m3.63980535s) [noise=true , address=145.79.227.48:4303 , piece_id=O7YGNBM7YSVRRPZSC523YTU22EOKZV37F2LNAQUJFKLOO6YDZUGQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.775448272 (618h55m3.646531317s) [noise=true , address=145.79.227.15:4876 , piece_id=NEN2X6QH2KISVRGSPB3A2H3WEXAQA5OPJO4NGRW5LXGHLBHHLOXQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.787493989 (618h55m3.658577044s) [noise=true , address=145.79.227.41:4086 , piece_id=VPJMKGMAAGAXL2VFQA27VUWY42S2J4HJXZD7SZ2TLRV6J6U2NW2Q, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.788437586 (618h55m3.659520621s) [noise=true , address=145.79.227.32:4836 , piece_id=MWM3HZG5XAS4QFOD2OBZMDT663HMTFZGWPCIM2XQVUTTI7YJBQ2A, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.791913014 (618h55m3.662996059s) [noise=true , address=145.79.227.45:4229 , piece_id=ZMNMN6EPBVAKHC4WICUKQIREOPVW3DIIFB6Z6PT4NUHFCFPDWTUA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.793094814 (618h55m3.664177879s) [noise=true , address=145.79.227.19:4078 , piece_id=7IYBWUPQV3WHQE7KHVQCTEBMSPS7VAPEZ2CKNBIMIQ2ELPOL5KOA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.794014435 (618h55m3.66509749s) [noise=true , address=145.79.227.47:4267 , piece_id=OQ6RYTP7ETYWECXVIYZFY24PL7UVSEO6PIQA6PDFUGYATVAS6NCA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.858627117 (618h55m3.729710163s) [noise=true , address=145.79.227.14:4837 , piece_id=CF3WPQBUBJ4GVNOPYTGYK6YVUDEW64V77MIZQM6ODUNQR2JKXUTA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:03.878179428 (618h55m3.749262473s) [noise=true , address=145.79.227.20:4438 , piece_id=FR5LTEP2QRIJDWMF3TMHDWJEW7UB6K6GO3BRGSHLPS627HOH4DPA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.198003230 (618h55m4.069086275s) [noise=true , address=45.91.116.9:4264 , piece_id=WSP4DN4EBC2NM6B6LHPFHC3E7NLULMFO2FCDGUVPN6XU3E766BRA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.413620056 (618h55m4.284703101s) [noise=true , address=77.90.31.14:4661 , piece_id=6SZUKP6NQ2D75OMAHF2CKQLE5UW64MROCTFO5S3ZQI4LVH5OKULQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.784331816 (618h55m4.655414861s) [noise=true , address=77.90.31.13:4629 , piece_id=7GZUZM4E4QNEJCWE2QCUBIO3RLVBGJVESMNMLRH4TB4I5JCWIYYA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: <nil> Dec 31 10:24:04.784357224 (618h55m4.655440299s) [seg_pos={0 0}, upload=542372]: Segment reached optimal threshold of 13 pieces. Dec 31 10:24:04.784615215 (618h55m4.65569826s) [noise=true , address=45.91.116.13:4153 , piece_id=KON2KEYYBP3C7SNETAZ3CWHT4PLZWIPDWCZKLROO7VZGOR6B3GJQ, seg_pos={0 0}, upload=542372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 10:24:04.784625245 (618h55m4.65570828s) [noise=true , address=45.91.116.13:4153 , piece_id=KON2KEYYBP3C7SNETAZ3CWHT4PLZWIPDWCZKLROO7VZGOR6B3GJQ, seg_pos={0 0}, upload=542372]: Long tail cancellation due to optimal threshold Dec 31 10:24:04.784889367 (618h55m4.655972402s) [noise=true , address=45.91.116.12:4227 , piece_id=A53DDZDX36E7ERSK5OPKMKNW3KSMZYUB2ARQHZJHEP3XGXVD2NUA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 31 10:24:04.784896462 (618h55m4.655979497s) [noise=true , address=45.91.116.12:4227 , piece_id=A53DDZDX36E7ERSK5OPKMKNW3KSMZYUB2ARQHZJHEP3XGXVD2NUA, seg_pos={0 0}, upload=542372]: Long tail cancellation due to optimal threshold Dec 31 10:24:04.784904366 (618h55m4.655987411s) [noise=true , address=77.90.31.10:4517 , piece_id=WGIZERW6GO6IQWSOO72SFVNYT47XEM36RHIOUWFM6EKQ4KXCMBGA, seg_pos={0 0}, upload=542372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 31 10:24:04.784936928 (618h55m4.656019973s) [noise=true , address=77.90.31.10:4517 , piece_id=WGIZERW6GO6IQWSOO72SFVNYT47XEM36RHIOUWFM6EKQ4KXCMBGA, seg_pos={0 0}, upload=542372]: Long tail cancellation due to optimal threshold Dec 31 10:24:04.784966214 (618h55m4.656049259s) [seg_pos={0 0}, upload=542372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:24:04.785146297 (618h55m4.656229342s) [upload=542372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:24:04.785154983 (618h55m4.656238028s) [upload=542372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:24:05.089489080 (618h55m4.960572126s) [upload=542372]: Done uploading segments... Dec 31 10:24:05.089515852 (618h55m4.960598897s) [upload=542372]: Upload finished. err: <nil> Duration: 30.003781566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X........X......X..............................] duration: 24.716233805s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:23:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 24.716261729s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X........XXX................XX......X....X.X...X..........X.] duration: 3.762139346s trace: https://jaeger.dev.storj.io/trace/68a4b215c6c93ce8 FAIL (2025-12-31 10:23:48 UTC) orbiter warning: context deadline exceeded Duration: 3.762180415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X........X......X.............................] duration: 19.04998513s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:23:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 19.050018913s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XX........XXX................XX......X....X.X...X..........X] duration: 5.000756911s trace: https://jaeger.dev.storj.io/trace/68a4b215c6c93ce8 FAIL (2025-12-31 10:23:17 UTC) orbiter warning: context deadline exceeded Duration: 5.000833366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX.] duration: 5.939859911s trace: https://jaeger.dev.storj.io/trace/4960953cf7105696 FAIL (2025-12-31 10:23:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 10:22:18.422304715 (618h53m18.293387761s) [upload=542347]: Starting upload Dec 31 10:22:18.422398364 (618h53m18.293481399s) [upload=542347]: Done starting upload Dec 31 10:22:18.422441296 (618h53m18.293524341s) [upload=542347]: Uploading segments... Dec 31 10:22:18.422654862 (618h53m18.293737907s) [upload=542347]: Got next segment. Inline: false Dec 31 10:22:18.422779800 (618h53m18.293862836s) [upload=542347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:22:18.422785862 (618h53m18.293868907s) [upload=542347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:22:18.667849120 (618h53m18.538932165s) [seg_pos={0 0}, upload=542347]: Begin upload segment... Dec 31 10:22:18.667898675 (618h53m18.53898172s) [seg_pos={0 0}, upload=542347]: Done begin upload segment. Dec 31 10:22:18.667926919 (618h53m18.539009964s) [noise=true , address=45.91.116.13:4155 , piece_id=OHZHXCASN7XYJ3ZC75WWRKBGBV2MWYY2LQFEVAYX7HIYZIVNVK7A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668002112 (618h53m18.539085147s) [noise=true , address=145.79.227.5:4221 , piece_id=7RA23EK6567HYSFPUWTFO2ETH24RVYULQQFZ4CJNQDQZWZXJXPZQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668450546 (618h53m18.539533591s) [noise=true , address=77.90.31.15:4700 , piece_id=GIGHBTOGZGUP26FT7M7GM45SDPDTTUGIERJVJXED426CIHLHUBBQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668541979 (618h53m18.539625014s) [noise=true , address=145.79.227.15:4875 , piece_id=MBDO4VMENFZCJ63J7PBBGRRPXE3HAQ3WQRGOP6NHZSKNEX7PIQPQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668821331 (618h53m18.539904366s) [noise=true , address=145.79.227.17:4970 , piece_id=73AW6PJBAC4M5P6PPSM2BH5ZV7R2FCJNHRRXTWJ3EWCT2Y3HTWSA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669092367 (618h53m18.540175402s) [noise=true , address=77.90.31.16:4736 , piece_id=HK2347U42ZDNN2MDIG3IPPLBLHACZRHOQLWX36KQ2CTHSRFPKBBA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669372561 (618h53m18.540455596s) [noise=true , address=77.90.31.11:4556 , piece_id=CS2DL6TEUFWATB7I3DIHC5VA7WJZ3T2763MMFJNMZC4CG6SIHBLQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669633919 (618h53m18.540716964s) [noise=true , address=145.79.227.46:4602 , piece_id=U3DCPET55KF7IWAW6HUDPFIFRJHEBCQ7ILDQDFMYKHIXV26YPKOA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669960911 (618h53m18.541043946s) [noise=true , address=45.91.116.14:4190 , piece_id=7ODOPE3UCYIPEFGCN6ZRZQR3TI2BYVEICX2OLAO6B3NJKY644G2A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670131917 (618h53m18.541214962s) [noise=true , address=145.79.227.43:4157 , piece_id=EAUJXKS6RGKRBXO3AK6ZNK67KAVFGXPXMIO4XMSZB4CCZXW4WE4A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670253258 (618h53m18.541336293s) [noise=true , address=145.79.227.50:4373 , piece_id=CRAIGVXJVXPEREMPUJUSNP3W2COESLZ4562IPHL3N7FN3S2WXN2A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670472376 (618h53m18.541555411s) [noise=true , address=77.90.31.14:4661 , piece_id=JNXMX4ALXFIK2MPOB3YFAM236OE3RALOPBIAFN6E7HJ5OYCV7YPQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670771264 (618h53m18.541854299s) [noise=true , address=45.91.116.11:4118 , piece_id=CGI4IQSOQTYZ2DNJHDJNOFPGQZ7TPRTCB7OOTWMIKO574MCZODQQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.671037041 (618h53m18.542120076s) [noise=true , address=45.91.116.12:4227 , piece_id=M5WHB3EBAOJ3EBXL3BQKEEFSZBQTATBFUC6UTZYY3LXOWUY3Q5BQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.671391906 (618h53m18.542474951s) [noise=true , address=145.79.227.3:4150 , piece_id=YYKK25URNGQVSJIOOSSSE7LZQBBFG6HYNFREVJNFBTXOOQ2MGXJA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.671740199 (618h53m18.542823234s) [noise=true , address=77.90.31.18:4805 , piece_id=K6DB6PRM6TVNOEUDKMVQL72ETX5QZ3DF5ZVZ67VJQERVCG6QXMMA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:20.376372096 (618h53m20.247455131s) [upload=542347]: Next returned nil segment Dec 31 10:22:20.376387505 (618h53m20.24747054s) [upload=542347]: Waiting for error group managing segments... Dec 31 10:22:20.660826922 (618h53m20.531909967s) [noise=true , address=145.79.227.17:4970 , piece_id=73AW6PJBAC4M5P6PPSM2BH5ZV7R2FCJNHRRXTWJ3EWCT2Y3HTWSA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.664438759 (618h53m20.535521804s) [noise=true , address=145.79.227.46:4602 , piece_id=U3DCPET55KF7IWAW6HUDPFIFRJHEBCQ7ILDQDFMYKHIXV26YPKOA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.672996374 (618h53m20.544079419s) [noise=true , address=145.79.227.5:4221 , piece_id=7RA23EK6567HYSFPUWTFO2ETH24RVYULQQFZ4CJNQDQZWZXJXPZQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.676248126 (618h53m20.547331171s) [noise=true , address=145.79.227.15:4875 , piece_id=MBDO4VMENFZCJ63J7PBBGRRPXE3HAQ3WQRGOP6NHZSKNEX7PIQPQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.680453534 (618h53m20.551536579s) [noise=true , address=145.79.227.3:4150 , piece_id=YYKK25URNGQVSJIOOSSSE7LZQBBFG6HYNFREVJNFBTXOOQ2MGXJA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.828645024 (618h53m20.699728079s) [noise=true , address=77.90.31.11:4556 , piece_id=CS2DL6TEUFWATB7I3DIHC5VA7WJZ3T2763MMFJNMZC4CG6SIHBLQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.837012157 (618h53m20.708095202s) [noise=true , address=77.90.31.16:4736 , piece_id=HK2347U42ZDNN2MDIG3IPPLBLHACZRHOQLWX36KQ2CTHSRFPKBBA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.933089072 (618h53m20.804172107s) [noise=true , address=77.90.31.15:4700 , piece_id=GIGHBTOGZGUP26FT7M7GM45SDPDTTUGIERJVJXED426CIHLHUBBQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.943415855 (618h53m20.8144989s) [noise=true , address=77.90.31.14:4661 , piece_id=JNXMX4ALXFIK2MPOB3YFAM236OE3RALOPBIAFN6E7HJ5OYCV7YPQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.943695678 (618h53m20.814778713s) [noise=true , address=77.90.31.18:4805 , piece_id=K6DB6PRM6TVNOEUDKMVQL72ETX5QZ3DF5ZVZ67VJQERVCG6QXMMA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.981022553 (618h53m20.852105588s) [noise=true , address=45.91.116.13:4155 , piece_id=OHZHXCASN7XYJ3ZC75WWRKBGBV2MWYY2LQFEVAYX7HIYZIVNVK7A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:21.099518568 (618h53m20.970601603s) [noise=true , address=45.91.116.12:4227 , piece_id=M5WHB3EBAOJ3EBXL3BQKEEFSZBQTATBFUC6UTZYY3LXOWUY3Q5BQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:21.195389030 (618h53m21.066472065s) [noise=true , address=45.91.116.14:4190 , piece_id=7ODOPE3UCYIPEFGCN6ZRZQR3TI2BYVEICX2OLAO6B3NJKY644G2A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:21.195410151 (618h53m21.066493196s) [seg_pos={0 0}, upload=542347]: Segment reached optimal threshold of 13 pieces. Dec 31 10:22:21.195637023 (618h53m21.066720068s) [noise=true , address=145.79.227.50:4373 , piece_id=CRAIGVXJVXPEREMPUJUSNP3W2COESLZ4562IPHL3N7FN3S2WXN2A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 31 10:22:21.195643896 (618h53m21.066726941s) [noise=true , address=145.79.227.50:4373 , piece_id=CRAIGVXJVXPEREMPUJUSNP3W2COESLZ4562IPHL3N7FN3S2WXN2A, seg_pos={0 0}, upload=542347]: Long tail cancellation due to optimal threshold Dec 31 10:22:21.195851371 (618h53m21.066934406s) [noise=true , address=45.91.116.11:4118 , piece_id=CGI4IQSOQTYZ2DNJHDJNOFPGQZ7TPRTCB7OOTWMIKO574MCZODQQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:40056->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:40056->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40056->45.91.116.11:4118: use of closed network connection Dec 31 10:22:21.195858234 (618h53m21.066941279s) [noise=true , address=45.91.116.11:4118 , piece_id=CGI4IQSOQTYZ2DNJHDJNOFPGQZ7TPRTCB7OOTWMIKO574MCZODQQ, seg_pos={0 0}, upload=542347]: Long tail cancellation due to optimal threshold Dec 31 10:22:21.195981669 (618h53m21.067064734s) [noise=true , address=145.79.227.43:4157 , piece_id=EAUJXKS6RGKRBXO3AK6ZNK67KAVFGXPXMIO4XMSZB4CCZXW4WE4A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 10:22:21.196015132 (618h53m21.067098167s) [noise=true , address=145.79.227.43:4157 , piece_id=EAUJXKS6RGKRBXO3AK6ZNK67KAVFGXPXMIO4XMSZB4CCZXW4WE4A, seg_pos={0 0}, upload=542347]: Long tail cancellation due to optimal threshold Dec 31 10:22:21.196056772 (618h53m21.067139807s) [seg_pos={0 0}, upload=542347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:22:21.196147555 (618h53m21.0672306s) [upload=542347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:22:21.196155049 (618h53m21.067238084s) [upload=542347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:22:21.491871420 (618h53m21.362954455s) [upload=542347]: Done uploading segments... Dec 31 10:22:21.491893061 (618h53m21.362976106s) [upload=542347]: Upload finished. err: <nil> Duration: 5.939902292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X.....X....XXX..X.X.X....XX........X.XX..XX......XXXXX] duration: 30.002580324s trace: https://jaeger.dev.storj.io/trace/4960953cf7105696 FAIL (2025-12-31 10:22:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 10:22:18.422304715 (618h53m18.293387761s) [upload=542347]: Starting upload Dec 31 10:22:18.422398364 (618h53m18.293481399s) [upload=542347]: Done starting upload Dec 31 10:22:18.422441296 (618h53m18.293524341s) [upload=542347]: Uploading segments... Dec 31 10:22:18.422654862 (618h53m18.293737907s) [upload=542347]: Got next segment. Inline: false Dec 31 10:22:18.422779800 (618h53m18.293862836s) [upload=542347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:22:18.422785862 (618h53m18.293868907s) [upload=542347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:22:18.667849120 (618h53m18.538932165s) [seg_pos={0 0}, upload=542347]: Begin upload segment... Dec 31 10:22:18.667898675 (618h53m18.53898172s) [seg_pos={0 0}, upload=542347]: Done begin upload segment. Dec 31 10:22:18.667926919 (618h53m18.539009964s) [noise=true , address=45.91.116.13:4155 , piece_id=OHZHXCASN7XYJ3ZC75WWRKBGBV2MWYY2LQFEVAYX7HIYZIVNVK7A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668002112 (618h53m18.539085147s) [noise=true , address=145.79.227.5:4221 , piece_id=7RA23EK6567HYSFPUWTFO2ETH24RVYULQQFZ4CJNQDQZWZXJXPZQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668450546 (618h53m18.539533591s) [noise=true , address=77.90.31.15:4700 , piece_id=GIGHBTOGZGUP26FT7M7GM45SDPDTTUGIERJVJXED426CIHLHUBBQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668541979 (618h53m18.539625014s) [noise=true , address=145.79.227.15:4875 , piece_id=MBDO4VMENFZCJ63J7PBBGRRPXE3HAQ3WQRGOP6NHZSKNEX7PIQPQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.668821331 (618h53m18.539904366s) [noise=true , address=145.79.227.17:4970 , piece_id=73AW6PJBAC4M5P6PPSM2BH5ZV7R2FCJNHRRXTWJ3EWCT2Y3HTWSA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669092367 (618h53m18.540175402s) [noise=true , address=77.90.31.16:4736 , piece_id=HK2347U42ZDNN2MDIG3IPPLBLHACZRHOQLWX36KQ2CTHSRFPKBBA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669372561 (618h53m18.540455596s) [noise=true , address=77.90.31.11:4556 , piece_id=CS2DL6TEUFWATB7I3DIHC5VA7WJZ3T2763MMFJNMZC4CG6SIHBLQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669633919 (618h53m18.540716964s) [noise=true , address=145.79.227.46:4602 , piece_id=U3DCPET55KF7IWAW6HUDPFIFRJHEBCQ7ILDQDFMYKHIXV26YPKOA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.669960911 (618h53m18.541043946s) [noise=true , address=45.91.116.14:4190 , piece_id=7ODOPE3UCYIPEFGCN6ZRZQR3TI2BYVEICX2OLAO6B3NJKY644G2A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670131917 (618h53m18.541214962s) [noise=true , address=145.79.227.43:4157 , piece_id=EAUJXKS6RGKRBXO3AK6ZNK67KAVFGXPXMIO4XMSZB4CCZXW4WE4A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670253258 (618h53m18.541336293s) [noise=true , address=145.79.227.50:4373 , piece_id=CRAIGVXJVXPEREMPUJUSNP3W2COESLZ4562IPHL3N7FN3S2WXN2A, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670472376 (618h53m18.541555411s) [noise=true , address=77.90.31.14:4661 , piece_id=JNXMX4ALXFIK2MPOB3YFAM236OE3RALOPBIAFN6E7HJ5OYCV7YPQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.670771264 (618h53m18.541854299s) [noise=true , address=45.91.116.11:4118 , piece_id=CGI4IQSOQTYZ2DNJHDJNOFPGQZ7TPRTCB7OOTWMIKO574MCZODQQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.671037041 (618h53m18.542120076s) [noise=true , address=45.91.116.12:4227 , piece_id=M5WHB3EBAOJ3EBXL3BQKEEFSZBQTATBFUC6UTZYY3LXOWUY3Q5BQ, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.671391906 (618h53m18.542474951s) [noise=true , address=145.79.227.3:4150 , piece_id=YYKK25URNGQVSJIOOSSSE7LZQBBFG6HYNFREVJNFBTXOOQ2MGXJA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:18.671740199 (618h53m18.542823234s) [noise=true , address=77.90.31.18:4805 , piece_id=K6DB6PRM6TVNOEUDKMVQL72ETX5QZ3DF5ZVZ67VJQERVCG6QXMMA, seg_pos={0 0}, upload=542347]: Uploading piece... Dec 31 10:22:20.376372096 (618h53m20.247455131s) [upload=542347]: Next returned nil segment Dec 31 10:22:20.376387505 (618h53m20.24747054s) [upload=542347]: Waiting for error group managing segments... Dec 31 10:22:20.660826922 (618h53m20.531909967s) [noise=true , address=145.79.227.17:4970 , piece_id=73AW6PJBAC4M5P6PPSM2BH5ZV7R2FCJNHRRXTWJ3EWCT2Y3HTWSA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.664438759 (618h53m20.535521804s) [noise=true , address=145.79.227.46:4602 , piece_id=U3DCPET55KF7IWAW6HUDPFIFRJHEBCQ7ILDQDFMYKHIXV26YPKOA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.672996374 (618h53m20.544079419s) [noise=true , address=145.79.227.5:4221 , piece_id=7RA23EK6567HYSFPUWTFO2ETH24RVYULQQFZ4CJNQDQZWZXJXPZQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.676248126 (618h53m20.547331171s) [noise=true , address=145.79.227.15:4875 , piece_id=MBDO4VMENFZCJ63J7PBBGRRPXE3HAQ3WQRGOP6NHZSKNEX7PIQPQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.680453534 (618h53m20.551536579s) [noise=true , address=145.79.227.3:4150 , piece_id=YYKK25URNGQVSJIOOSSSE7LZQBBFG6HYNFREVJNFBTXOOQ2MGXJA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.828645024 (618h53m20.699728079s) [noise=true , address=77.90.31.11:4556 , piece_id=CS2DL6TEUFWATB7I3DIHC5VA7WJZ3T2763MMFJNMZC4CG6SIHBLQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.837012157 (618h53m20.708095202s) [noise=true , address=77.90.31.16:4736 , piece_id=HK2347U42ZDNN2MDIG3IPPLBLHACZRHOQLWX36KQ2CTHSRFPKBBA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.933089072 (618h53m20.804172107s) [noise=true , address=77.90.31.15:4700 , piece_id=GIGHBTOGZGUP26FT7M7GM45SDPDTTUGIERJVJXED426CIHLHUBBQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.943415855 (618h53m20.8144989s) [noise=true , address=77.90.31.14:4661 , piece_id=JNXMX4ALXFIK2MPOB3YFAM236OE3RALOPBIAFN6E7HJ5OYCV7YPQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.943695678 (618h53m20.814778713s) [noise=true , address=77.90.31.18:4805 , piece_id=K6DB6PRM6TVNOEUDKMVQL72ETX5QZ3DF5ZVZ67VJQERVCG6QXMMA, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:20.981022553 (618h53m20.852105588s) [noise=true , address=45.91.116.13:4155 , piece_id=OHZHXCASN7XYJ3ZC75WWRKBGBV2MWYY2LQFEVAYX7HIYZIVNVK7A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:21.099518568 (618h53m20.970601603s) [noise=true , address=45.91.116.12:4227 , piece_id=M5WHB3EBAOJ3EBXL3BQKEEFSZBQTATBFUC6UTZYY3LXOWUY3Q5BQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:21.195389030 (618h53m21.066472065s) [noise=true , address=45.91.116.14:4190 , piece_id=7ODOPE3UCYIPEFGCN6ZRZQR3TI2BYVEICX2OLAO6B3NJKY644G2A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: <nil> Dec 31 10:22:21.195410151 (618h53m21.066493196s) [seg_pos={0 0}, upload=542347]: Segment reached optimal threshold of 13 pieces. Dec 31 10:22:21.195637023 (618h53m21.066720068s) [noise=true , address=145.79.227.50:4373 , piece_id=CRAIGVXJVXPEREMPUJUSNP3W2COESLZ4562IPHL3N7FN3S2WXN2A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 31 10:22:21.195643896 (618h53m21.066726941s) [noise=true , address=145.79.227.50:4373 , piece_id=CRAIGVXJVXPEREMPUJUSNP3W2COESLZ4562IPHL3N7FN3S2WXN2A, seg_pos={0 0}, upload=542347]: Long tail cancellation due to optimal threshold Dec 31 10:22:21.195851371 (618h53m21.066934406s) [noise=true , address=45.91.116.11:4118 , piece_id=CGI4IQSOQTYZ2DNJHDJNOFPGQZ7TPRTCB7OOTWMIKO574MCZODQQ, seg_pos={0 0}, upload=542347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:40056->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:40056->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40056->45.91.116.11:4118: use of closed network connection Dec 31 10:22:21.195858234 (618h53m21.066941279s) [noise=true , address=45.91.116.11:4118 , piece_id=CGI4IQSOQTYZ2DNJHDJNOFPGQZ7TPRTCB7OOTWMIKO574MCZODQQ, seg_pos={0 0}, upload=542347]: Long tail cancellation due to optimal threshold Dec 31 10:22:21.195981669 (618h53m21.067064734s) [noise=true , address=145.79.227.43:4157 , piece_id=EAUJXKS6RGKRBXO3AK6ZNK67KAVFGXPXMIO4XMSZB4CCZXW4WE4A, seg_pos={0 0}, upload=542347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 10:22:21.196015132 (618h53m21.067098167s) [noise=true , address=145.79.227.43:4157 , piece_id=EAUJXKS6RGKRBXO3AK6ZNK67KAVFGXPXMIO4XMSZB4CCZXW4WE4A, seg_pos={0 0}, upload=542347]: Long tail cancellation due to optimal threshold Dec 31 10:22:21.196056772 (618h53m21.067139807s) [seg_pos={0 0}, upload=542347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:22:21.196147555 (618h53m21.0672306s) [upload=542347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:22:21.196155049 (618h53m21.067238084s) [upload=542347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:22:21.491871420 (618h53m21.362954455s) [upload=542347]: Done uploading segments... Dec 31 10:22:21.491893061 (618h53m21.362976106s) [upload=542347]: Upload finished. err: <nil> Duration: 30.002666137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......X.....X....XXX..X.X.X....XX........X.XX..XX......XXXX] duration: 30.002688655s trace: https://jaeger.dev.storj.io/trace/53c6a74eccc6a112 FAIL (2025-12-31 10:22:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: ZZ5XSVUYTEKMNX2J3FTKDR33ANZ3BBW4J6RZ4RMYLCO55QKBMXVQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2GDUEFFUF6RAWO625WSBUCCWSK26VI4LC3LDSAPHPAIQPRODTGQQ) context deadline exceeded ======LOGS====== egments... Dec 31 10:21:48.194448919 (618h52m48.065531964s) [upload=542337]: Got next segment. Inline: false Dec 31 10:21:48.194487693 (618h52m48.065570728s) [upload=542337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:21:48.194493554 (618h52m48.065576599s) [upload=542337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:21:48.436027342 (618h52m48.307110397s) [seg_pos={0 0}, upload=542337]: Begin upload segment... Dec 31 10:21:48.436074622 (618h52m48.307157667s) [seg_pos={0 0}, upload=542337]: Done begin upload segment. Dec 31 10:21:48.436084922 (618h52m48.307167957s) [noise=true , address=145.79.227.16:4911 , piece_id=4VUOG6JQD5VPD633NZCGRPNBXIUZHJGEC7ZGH75XPQUM7ONPXMQA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.436107635 (618h52m48.30719068s) [noise=true , address=145.79.227.48:4302 , piece_id=54YD7AWQPLP5PW3RE5LSQ2VRV4BVDMYC7SM2DJIFZXXVMGJHEDJQ, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.436715853 (618h52m48.307798928s) [noise=true , address=145.79.227.9:4366 , piece_id=UAICJVLCOVNQLQUQG6KA3FUUFILH3CCB227WM6M7NER3UQWQP54Q, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.437121616 (618h52m48.308204661s) [noise=true , address=145.79.227.37:4566 , piece_id=LZONFFTDPBAAUQAYWPZXK6XMCWJ5Q2PLC6IHM42NKEDPMDAI4GCA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.437753188 (618h52m48.308836233s) [noise=true , address=145.79.227.44:4193 , piece_id=IHTHXDOVBL2F2HEF3PJCEDFBAJLIRKBREOAEFL2HP5JPQHXZAJBA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.438193738 (618h52m48.309276783s) [noise=true , address=145.79.227.27:4691 , piece_id=YDX5R2EFS2CXQ3GRQ5GLOCGGGD4YV3GTQLJSNH5UP65PJIM5IIMQ, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.438598137 (618h52m48.309681172s) [noise=true , address=145.79.227.3:4151 , piece_id=IQDGDMTDHLYWHG7KGDATVVQQXOAK4TCBSK6JWIFSINXZRDLKLA7Q, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.439006084 (618h52m48.310089119s) [noise=true , address=145.79.227.18:4006 , piece_id=JF7KJIA6AGC23Y2H7YBVDOJ6FE7LW6RJOF53IIBO4CIVU7MJWKDA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.439466521 (618h52m48.310549556s) [noise=true , address=45.91.116.8:4047 , piece_id=EYJ3UEEUSMUQQJI5EGWCZN6QZNHZBW27DENJ23B7DBP7YKTIB7TA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.439830194 (618h52m48.310913229s) [noise=true , address=145.79.227.36:4517 , piece_id=YYNK5NVPZ6V2PX5JMXZ7A564JL3PNDPAOPTSAEXAK64SCMYV6ZKA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.439966674 (618h52m48.311049719s) [noise=true , address=45.91.116.11:4117 , piece_id=KIAEEOJ2LUJMKMP7B2BAXYZFW2QZ5KPUEJAMHSD3T4QKVX2HQYJQ, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.440455635 (618h52m48.31153867s) [noise=true , address=145.79.227.15:4873 , piece_id=6UEFKPWEZ7WGGF3QZXBZY2KLQGD7PER7PX43X4WLZF7XVODERV4A, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.440890091 (618h52m48.311973126s) [noise=true , address=77.90.31.13:4629 , piece_id=2GDUEFFUF6RAWO625WSBUCCWSK26VI4LC3LDSAPHPAIQPRODTGQQ, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.441141780 (618h52m48.312224815s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZ5XSVUYTEKMNX2J3FTKDR33ANZ3BBW4J6RZ4RMYLCO55QKBMXVQ, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.441616274 (618h52m48.312699309s) [noise=true , address=145.79.227.11:4731 , piece_id=K4PODXQF3AS7DWHUYCKVNM5IQU4TAKBVLBLSRHF5BGCHM6GZFKWA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:48.442083443 (618h52m48.313166489s) [noise=true , address=145.79.227.10:4403 , piece_id=OQYKRYPLSS76OXFEFG4FGIRT6OA4GO6NHXD2IMPYORRAHMH5PCIA, seg_pos={0 0}, upload=542337]: Uploading piece... Dec 31 10:21:50.128306225 (618h52m49.99938927s) [upload=542337]: Next returned nil segment Dec 31 10:21:50.128321093 (618h52m49.999404128s) [upload=542337]: Waiting for error group managing segments... Dec 31 10:21:50.413200999 (618h52m50.284284034s) [noise=true , address=145.79.227.9:4366 , piece_id=UAICJVLCOVNQLQUQG6KA3FUUFILH3CCB227WM6M7NER3UQWQP54Q, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.422384626 (618h52m50.293467661s) [noise=true , address=145.79.227.18:4006 , piece_id=JF7KJIA6AGC23Y2H7YBVDOJ6FE7LW6RJOF53IIBO4CIVU7MJWKDA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.426221883 (618h52m50.297304918s) [noise=true , address=145.79.227.15:4873 , piece_id=6UEFKPWEZ7WGGF3QZXBZY2KLQGD7PER7PX43X4WLZF7XVODERV4A, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.426785275 (618h52m50.29786832s) [noise=true , address=145.79.227.3:4151 , piece_id=IQDGDMTDHLYWHG7KGDATVVQQXOAK4TCBSK6JWIFSINXZRDLKLA7Q, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.428080101 (618h52m50.299163146s) [noise=true , address=145.79.227.27:4691 , piece_id=YDX5R2EFS2CXQ3GRQ5GLOCGGGD4YV3GTQLJSNH5UP65PJIM5IIMQ, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.435323002 (618h52m50.306406037s) [noise=true , address=145.79.227.37:4566 , piece_id=LZONFFTDPBAAUQAYWPZXK6XMCWJ5Q2PLC6IHM42NKEDPMDAI4GCA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.442515539 (618h52m50.313598574s) [noise=true , address=145.79.227.10:4403 , piece_id=OQYKRYPLSS76OXFEFG4FGIRT6OA4GO6NHXD2IMPYORRAHMH5PCIA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.445686967 (618h52m50.316770013s) [noise=true , address=145.79.227.11:4731 , piece_id=K4PODXQF3AS7DWHUYCKVNM5IQU4TAKBVLBLSRHF5BGCHM6GZFKWA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.450159513 (618h52m50.321242549s) [noise=true , address=145.79.227.48:4302 , piece_id=54YD7AWQPLP5PW3RE5LSQ2VRV4BVDMYC7SM2DJIFZXXVMGJHEDJQ, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.456677666 (618h52m50.327760711s) [noise=true , address=145.79.227.36:4517 , piece_id=YYNK5NVPZ6V2PX5JMXZ7A564JL3PNDPAOPTSAEXAK64SCMYV6ZKA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.513358254 (618h52m50.384441289s) [noise=true , address=145.79.227.16:4911 , piece_id=4VUOG6JQD5VPD633NZCGRPNBXIUZHJGEC7ZGH75XPQUM7ONPXMQA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.633717356 (618h52m50.504800391s) [noise=true , address=77.90.31.13:4629 , piece_id=2GDUEFFUF6RAWO625WSBUCCWSK26VI4LC3LDSAPHPAIQPRODTGQQ, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.734250357 (618h52m50.605333393s) [noise=true , address=77.90.31.12:4590 , piece_id=ZZ5XSVUYTEKMNX2J3FTKDR33ANZ3BBW4J6RZ4RMYLCO55QKBMXVQ, seg_pos={0 0}, upload=542337]: Done uploading piece. err: <nil> Dec 31 10:21:50.734269263 (618h52m50.605352298s) [seg_pos={0 0}, upload=542337]: Segment reached optimal threshold of 13 pieces. Dec 31 10:21:50.734436021 (618h52m50.605519056s) [noise=true , address=45.91.116.11:4117 , piece_id=KIAEEOJ2LUJMKMP7B2BAXYZFW2QZ5KPUEJAMHSD3T4QKVX2HQYJQ, seg_pos={0 0}, upload=542337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:41568->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:41568->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41568->45.91.116.11:4117: use of closed network connection Dec 31 10:21:50.734441030 (618h52m50.605524065s) [noise=true , address=45.91.116.11:4117 , piece_id=KIAEEOJ2LUJMKMP7B2BAXYZFW2QZ5KPUEJAMHSD3T4QKVX2HQYJQ, seg_pos={0 0}, upload=542337]: Long tail cancellation due to optimal threshold Dec 31 10:21:50.734502327 (618h52m50.605585382s) [noise=true , address=45.91.116.8:4047 , piece_id=EYJ3UEEUSMUQQJI5EGWCZN6QZNHZBW27DENJ23B7DBP7YKTIB7TA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:50410->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:50410->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50410->45.91.116.8:4047: use of closed network connection Dec 31 10:21:50.734513227 (618h52m50.605596273s) [noise=true , address=45.91.116.8:4047 , piece_id=EYJ3UEEUSMUQQJI5EGWCZN6QZNHZBW27DENJ23B7DBP7YKTIB7TA, seg_pos={0 0}, upload=542337]: Long tail cancellation due to optimal threshold Dec 31 10:21:50.734571148 (618h52m50.605654183s) [noise=true , address=145.79.227.44:4193 , piece_id=IHTHXDOVBL2F2HEF3PJCEDFBAJLIRKBREOAEFL2HP5JPQHXZAJBA, seg_pos={0 0}, upload=542337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 31 10:21:50.734576218 (618h52m50.605659253s) [noise=true , address=145.79.227.44:4193 , piece_id=IHTHXDOVBL2F2HEF3PJCEDFBAJLIRKBREOAEFL2HP5JPQHXZAJBA, seg_pos={0 0}, upload=542337]: Long tail cancellation due to optimal threshold Dec 31 10:21:50.734639458 (618h52m50.605722503s) [seg_pos={0 0}, upload=542337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:21:50.734705664 (618h52m50.605788699s) [upload=542337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:21:50.734710323 (618h52m50.605793358s) [upload=542337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:21:51.016648006 (618h52m50.887731051s) [upload=542337]: Done uploading segments... Dec 31 10:21:51.016668405 (618h52m50.88775145s) [upload=542337]: Upload finished. err: <nil> Duration: 30.002716921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [XXX........XXX................XX......X....X.X...X..........] duration: 3.558717173s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:22:15 UTC) orbiter warning: context deadline exceeded Duration: 3.558744575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X........X......X............................] duration: 15.549778366s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:22:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 15.549803714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........X.....X....XXX..X.X.X....XX........X.XX..XX......XXX] duration: 30.004049686s trace: https://jaeger.dev.storj.io/trace/3a3b801b115a0ed9 FAIL (2025-12-31 10:21:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 10:21:12.879884647 (618h52m12.750967682s) [upload=542329]: Starting upload Dec 31 10:21:12.879947305 (618h52m12.75103034s) [upload=542329]: Done starting upload Dec 31 10:21:12.880015305 (618h52m12.75109834s) [upload=542329]: Uploading segments... Dec 31 10:21:12.880134512 (618h52m12.751217577s) [upload=542329]: Got next segment. Inline: false Dec 31 10:21:12.880186010 (618h52m12.751269055s) [upload=542329]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:21:12.880194517 (618h52m12.751277552s) [upload=542329]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:21:13.127210053 (618h52m12.998293108s) [seg_pos={0 0}, upload=542329]: Begin upload segment... Dec 31 10:21:13.127264947 (618h52m12.998347982s) [seg_pos={0 0}, upload=542329]: Done begin upload segment. Dec 31 10:21:13.127298461 (618h52m12.998381506s) [noise=true , address=145.79.227.39:4013 , piece_id=R5I7G32SY24S4OZJOXKVQ64L6NUNXM3Q6U4EOZ4J7ZMGXB4XDT7Q, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.127390807 (618h52m12.998473842s) [noise=true , address=145.79.227.11:4731 , piece_id=X3GUECULVDUU7X5RYWGJK4MRLU443NMG23IU7TXD3DHN5LMQ227Q, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.128102211 (618h52m12.999185256s) [noise=true , address=45.91.116.12:4228 , piece_id=X6MWGFWOVS73KUHIFW37T2K6JHKKFGL4SH4BE5C5BMBXOGJTWQFA, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.128438562 (618h52m12.999521637s) [noise=true , address=145.79.227.32:4836 , piece_id=JWHUHEHRDKR76BL26LBHRZ5MICPMT5V67G2F3T42IAPELR4XIGSQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.128810671 (618h52m12.999893706s) [noise=true , address=77.90.31.12:4590 , piece_id=MWYBD4IBVU34GN6G4LDCE6F3GRGK25RKE6FZJF2O2V4L5UXUXR4Q, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.129195283 (618h52m13.000278318s) [noise=true , address=145.79.227.15:4873 , piece_id=R3FVUYO6M7PDY3SBMDC4UIVVGQ54YXAZS5UFR66UOKMCCN3IJYPA, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.129535099 (618h52m13.000618145s) [noise=true , address=77.90.31.17:4773 , piece_id=4O6FEVPQP5BQ2AJ4ZO4D7UXYSPZZJPS2CYU57IHGGM2VJSF6PHHQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.129986729 (618h52m13.001069775s) [noise=true , address=145.79.227.7:4295 , piece_id=46YLWW47WMRSPKRHSUXPJDASHWGWFIEXWN4V3LMUQOM2KJJFYFCQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.130377154 (618h52m13.001460199s) [noise=true , address=145.79.227.43:4157 , piece_id=YODF2EIECD3NUMUEBSWQ3M3PLUU64MDT66FGPF3F3UO77EX6NBAQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.132104071 (618h52m13.003187106s) [noise=true , address=77.90.31.18:4808 , piece_id=MTPVDCYXUU3OOZ34DF2CULKUN6WYW2YDLZMDMEK7ANMD5EX4RAFQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.134921236 (618h52m13.006004271s) [noise=true , address=77.90.31.11:4555 , piece_id=3RN4X7YUE4IDNKKTWSG45XGG3552PC7PYWQHKJVOT2DIP4YMT4MQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.138303987 (618h52m13.009387032s) [noise=true , address=145.79.227.17:4970 , piece_id=IHK4LK54KSMOD333M4HKSKVLML7D3OLAAH2H55HHFMB7BEXGFAQA, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.141667552 (618h52m13.012750597s) [noise=true , address=45.91.116.5:4297 , piece_id=CIELU6WJ2K575WRXPXHF7YW7XMV5L37LSUKQNRHSLZTRZMQ3ZAEQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.143261227 (618h52m13.014344262s) [noise=true , address=145.79.227.9:4365 , piece_id=RT2XMGPW6HLOZ2ECQPUGJP7R4SQXCXQVW7GMIAPABWQVECIFWPCQ, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.146560759 (618h52m13.017643794s) [noise=true , address=145.79.227.22:4511 , piece_id=PYBRXZAV4VJCSIRR3L5W3MN4ND3XZTEKHXZPVH6J2S2T3TQ5PR2Q, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:13.148257711 (618h52m13.019340746s) [noise=true , address=77.90.31.10:4517 , piece_id=65RFTOYRZUZBOVRHHK7OR37NQ3T7UMZMMHIPX5A6T7GPI52GTBIA, seg_pos={0 0}, upload=542329]: Uploading piece... Dec 31 10:21:14.842952344 (618h52m14.714035379s) [upload=542329]: Next returned nil segment Dec 31 10:21:14.842970068 (618h52m14.714053113s) [upload=542329]: Waiting for error group managing segments... Dec 31 10:21:15.114575618 (618h52m14.985658663s) [noise=true , address=145.79.227.11:4731 , piece_id=X3GUECULVDUU7X5RYWGJK4MRLU443NMG23IU7TXD3DHN5LMQ227Q, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.117613362 (618h52m14.988696407s) [noise=true , address=145.79.227.7:4295 , piece_id=46YLWW47WMRSPKRHSUXPJDASHWGWFIEXWN4V3LMUQOM2KJJFYFCQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.130995694 (618h52m15.002078729s) [noise=true , address=145.79.227.15:4873 , piece_id=R3FVUYO6M7PDY3SBMDC4UIVVGQ54YXAZS5UFR66UOKMCCN3IJYPA, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.144453490 (618h52m15.015536535s) [noise=true , address=145.79.227.22:4511 , piece_id=PYBRXZAV4VJCSIRR3L5W3MN4ND3XZTEKHXZPVH6J2S2T3TQ5PR2Q, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.145101082 (618h52m15.016184138s) [noise=true , address=145.79.227.32:4836 , piece_id=JWHUHEHRDKR76BL26LBHRZ5MICPMT5V67G2F3T42IAPELR4XIGSQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.145640079 (618h52m15.016723124s) [noise=true , address=145.79.227.9:4365 , piece_id=RT2XMGPW6HLOZ2ECQPUGJP7R4SQXCXQVW7GMIAPABWQVECIFWPCQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.146238789 (618h52m15.017321834s) [noise=true , address=145.79.227.39:4013 , piece_id=R5I7G32SY24S4OZJOXKVQ64L6NUNXM3Q6U4EOZ4J7ZMGXB4XDT7Q, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.156464070 (618h52m15.027547125s) [noise=true , address=145.79.227.17:4970 , piece_id=IHK4LK54KSMOD333M4HKSKVLML7D3OLAAH2H55HHFMB7BEXGFAQA, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.283496009 (618h52m15.154579044s) [noise=true , address=77.90.31.12:4590 , piece_id=MWYBD4IBVU34GN6G4LDCE6F3GRGK25RKE6FZJF2O2V4L5UXUXR4Q, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.375391472 (618h52m15.246474517s) [noise=true , address=77.90.31.18:4808 , piece_id=MTPVDCYXUU3OOZ34DF2CULKUN6WYW2YDLZMDMEK7ANMD5EX4RAFQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.424167639 (618h52m15.295250674s) [noise=true , address=77.90.31.17:4773 , piece_id=4O6FEVPQP5BQ2AJ4ZO4D7UXYSPZZJPS2CYU57IHGGM2VJSF6PHHQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.429024868 (618h52m15.300107903s) [noise=true , address=77.90.31.11:4555 , piece_id=3RN4X7YUE4IDNKKTWSG45XGG3552PC7PYWQHKJVOT2DIP4YMT4MQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.457792376 (618h52m15.328875421s) [noise=true , address=45.91.116.12:4228 , piece_id=X6MWGFWOVS73KUHIFW37T2K6JHKKFGL4SH4BE5C5BMBXOGJTWQFA, seg_pos={0 0}, upload=542329]: Done uploading piece. err: <nil> Dec 31 10:21:15.457848162 (618h52m15.328931207s) [seg_pos={0 0}, upload=542329]: Segment reached optimal threshold of 13 pieces. Dec 31 10:21:15.458130870 (618h52m15.329213915s) [noise=true , address=77.90.31.10:4517 , piece_id=65RFTOYRZUZBOVRHHK7OR37NQ3T7UMZMMHIPX5A6T7GPI52GTBIA, seg_pos={0 0}, upload=542329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:39410->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:39410->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39410->77.90.31.10:4517: use of closed network connection Dec 31 10:21:15.458140609 (618h52m15.329223654s) [noise=true , address=77.90.31.10:4517 , piece_id=65RFTOYRZUZBOVRHHK7OR37NQ3T7UMZMMHIPX5A6T7GPI52GTBIA, seg_pos={0 0}, upload=542329]: Long tail cancellation due to optimal threshold Dec 31 10:21:15.458190223 (618h52m15.329273278s) [noise=true , address=145.79.227.43:4157 , piece_id=YODF2EIECD3NUMUEBSWQ3M3PLUU64MDT66FGPF3F3UO77EX6NBAQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 31 10:21:15.458241550 (618h52m15.329324596s) [noise=true , address=145.79.227.43:4157 , piece_id=YODF2EIECD3NUMUEBSWQ3M3PLUU64MDT66FGPF3F3UO77EX6NBAQ, seg_pos={0 0}, upload=542329]: Long tail cancellation due to optimal threshold Dec 31 10:21:15.458527305 (618h52m15.32961035s) [noise=true , address=45.91.116.5:4297 , piece_id=CIELU6WJ2K575WRXPXHF7YW7XMV5L37LSUKQNRHSLZTRZMQ3ZAEQ, seg_pos={0 0}, upload=542329]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: context canceled; context canceled Dec 31 10:21:15.458537104 (618h52m15.329620139s) [noise=true , address=45.91.116.5:4297 , piece_id=CIELU6WJ2K575WRXPXHF7YW7XMV5L37LSUKQNRHSLZTRZMQ3ZAEQ, seg_pos={0 0}, upload=542329]: Long tail cancellation due to optimal threshold Dec 31 10:21:15.458567402 (618h52m15.329650437s) [seg_pos={0 0}, upload=542329]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:21:15.458652152 (618h52m15.329735198s) [upload=542329]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:21:15.458658905 (618h52m15.32974194s) [upload=542329]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:21:15.749736684 (618h52m15.620819729s) [upload=542329]: Done uploading segments... Dec 31 10:21:15.749769786 (618h52m15.620852831s) [upload=542329]: Upload finished. err: <nil> Duration: 30.004072058s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.XXX........XXX................XX......X....X.X...X.........] duration: 3.450479315s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:21:36 UTC) orbiter warning: context deadline exceeded Duration: 3.450504623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X........X......X...........................] duration: 15.370042742s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:21:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 15.370061657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X........X......X..........................] duration: 33.240050658s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:21:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 33.240077299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..XXX........XXX................XX......X....X.X...X........] duration: 3.692635543s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:20:34 UTC) orbiter warning: context deadline exceeded Duration: 3.692682492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........X.....X....XXX..X.X.X....XX........X.XX..XX......XX] duration: 30.003557146s trace: https://jaeger.dev.storj.io/trace/36e8031d295fc7dc FAIL (2025-12-31 10:20:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: BWYR3KRDC65CEGYLKHWHLDKE5V7M4UNHZDS3JLBN5NWM4I26OVTQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: AYN4RJZLNOMUAY4QSJNI7QAT27S5TRG34S7GXRGC7GOKWEMAPTPA) context deadline exceeded ======LOGS====== Dec 31 10:19:58.819761028 (618h50m58.690844063s) [upload=542312]: Starting upload Dec 31 10:19:58.819830390 (618h50m58.690913426s) [upload=542312]: Done starting upload Dec 31 10:19:58.819865107 (618h50m58.690948162s) [upload=542312]: Uploading segments... Dec 31 10:19:58.820004773 (618h50m58.691087808s) [upload=542312]: Got next segment. Inline: false Dec 31 10:19:58.820030602 (618h50m58.691113647s) [upload=542312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:19:58.820034559 (618h50m58.691117594s) [upload=542312]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:19:59.078475023 (618h50m58.949558058s) [seg_pos={0 0}, upload=542312]: Begin upload segment... Dec 31 10:19:59.078517473 (618h50m58.949600528s) [seg_pos={0 0}, upload=542312]: Done begin upload segment. Dec 31 10:19:59.078554604 (618h50m58.949637649s) [noise=true , address=145.79.227.13:4803 , piece_id=TKDEKP2KMM67AZJ4UIOVCPDYAJGV2YCMRAXVAV63H64ASUJB5ZDQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.078561006 (618h50m58.949644041s) [noise=true , address=145.79.227.39:4015 , piece_id=CBUKLPUMRL2QHQT2OICMXJGPPBHG3FYHY3IXBPZFS22X3OUPNXDA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.078612595 (618h50m58.94969564s) [noise=true , address=145.79.227.16:4910 , piece_id=OYBCSV7FRUTBO6TQQKEZWS534JRMVGIE4NMXKNS3VTOPVYZPARHQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.079132475 (618h50m58.95021551s) [noise=true , address=145.79.227.46:4602 , piece_id=HZB3AB37PMNR3NLMSCKN3ZL3A7A33NLMT3RYDPEHKDTYD7SJIZBQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.079709593 (618h50m58.950792628s) [noise=true , address=45.91.116.13:4153 , piece_id=4NQ24M5CMKYN6SUAT7FMGBGDNOYI4Z6VY4B5AIRRYUS6MGSUBYRA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.080043840 (618h50m58.951126885s) [noise=true , address=145.79.227.43:4157 , piece_id=WTYZSI65XN2T4LHKNPR3WPTQNOKXIQB3M547MICYW55OMHJNYVZQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.080409946 (618h50m58.951492981s) [noise=true , address=45.91.116.12:4225 , piece_id=BWYR3KRDC65CEGYLKHWHLDKE5V7M4UNHZDS3JLBN5NWM4I26OVTQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.080701803 (618h50m58.951784838s) [noise=true , address=145.79.227.18:4007 , piece_id=HA3ELFWD3UKFLSSPGJ6I5RAZEAQS54PFEMAJMACHOL63JBLVQJGQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.081019908 (618h50m58.952102953s) [noise=true , address=45.91.116.10:4081 , piece_id=GK2ZYPBFP24JQZK24HX4DMNRB2Z55EODPO6XYFM52FQK7TBY4XUQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.081413728 (618h50m58.952496763s) [noise=true , address=145.79.227.48:4303 , piece_id=GYCGCZ4EWLSQOLSDBBJHLKDNISRSQRJMFAZ7N5M74QGLPVBHXGJA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.082862106 (618h50m58.953945151s) [noise=true , address=145.79.227.2:4115 , piece_id=46TLR2L4P63IE44WEON5NZS4M2YDRDXCISHIS7WL4CR3JWFEDXMA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.084208971 (618h50m58.955292006s) [noise=true , address=45.91.116.14:4192 , piece_id=AYN4RJZLNOMUAY4QSJNI7QAT27S5TRG34S7GXRGC7GOKWEMAPTPA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.085782287 (618h50m58.956865322s) [noise=true , address=45.91.116.8:4047 , piece_id=JT72P3DFK6KHUPXI6SRJLWQQJYKPDSBF3HFSAOW6K7SDSIF3PVHQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.087234543 (618h50m58.958317588s) [noise=true , address=145.79.227.52:4445 , piece_id=D5AD4UDM6SVXVHIQRMCTXALZMDKWQME42HIZDZL2GRTN7FNNB4QQ, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.088363963 (618h50m58.959446998s) [noise=true , address=145.79.227.10:4401 , piece_id=TIZSAHYZJNP4OZSJQ4ZXXVPBGO46ONQ7VSNADVO33J6UVOJTRBXA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:19:59.088933777 (618h50m58.960016812s) [noise=true , address=145.79.227.49:4337 , piece_id=GKC5X2PJN2HZTRI6QXB6TTJMFJQE5JRKY4AY3USVH2NK4Z5GG6ZA, seg_pos={0 0}, upload=542312]: Uploading piece... Dec 31 10:20:00.701199701 (618h51m0.572282746s) [upload=542312]: Next returned nil segment Dec 31 10:20:00.701215781 (618h51m0.572298826s) [upload=542312]: Waiting for error group managing segments... Dec 31 10:20:01.068054299 (618h51m0.939137344s) [noise=true , address=145.79.227.52:4445 , piece_id=D5AD4UDM6SVXVHIQRMCTXALZMDKWQME42HIZDZL2GRTN7FNNB4QQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.070404394 (618h51m0.941487429s) [noise=true , address=145.79.227.48:4303 , piece_id=GYCGCZ4EWLSQOLSDBBJHLKDNISRSQRJMFAZ7N5M74QGLPVBHXGJA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.071460184 (618h51m0.942543229s) [noise=true , address=145.79.227.18:4007 , piece_id=HA3ELFWD3UKFLSSPGJ6I5RAZEAQS54PFEMAJMACHOL63JBLVQJGQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.073317681 (618h51m0.944400726s) [noise=true , address=145.79.227.2:4115 , piece_id=46TLR2L4P63IE44WEON5NZS4M2YDRDXCISHIS7WL4CR3JWFEDXMA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.078096982 (618h51m0.949180017s) [noise=true , address=145.79.227.16:4910 , piece_id=OYBCSV7FRUTBO6TQQKEZWS534JRMVGIE4NMXKNS3VTOPVYZPARHQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.084142844 (618h51m0.955225889s) [noise=true , address=145.79.227.46:4602 , piece_id=HZB3AB37PMNR3NLMSCKN3ZL3A7A33NLMT3RYDPEHKDTYD7SJIZBQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.092998256 (618h51m0.964081301s) [noise=true , address=145.79.227.39:4015 , piece_id=CBUKLPUMRL2QHQT2OICMXJGPPBHG3FYHY3IXBPZFS22X3OUPNXDA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.094065108 (618h51m0.965148143s) [noise=true , address=145.79.227.13:4803 , piece_id=TKDEKP2KMM67AZJ4UIOVCPDYAJGV2YCMRAXVAV63H64ASUJB5ZDQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.145771735 (618h51m1.01685478s) [noise=true , address=145.79.227.10:4401 , piece_id=TIZSAHYZJNP4OZSJQ4ZXXVPBGO46ONQ7VSNADVO33J6UVOJTRBXA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.171574517 (618h51m1.042657552s) [noise=true , address=145.79.227.43:4157 , piece_id=WTYZSI65XN2T4LHKNPR3WPTQNOKXIQB3M547MICYW55OMHJNYVZQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.284412900 (618h51m1.155495945s) [noise=true , address=145.79.227.49:4337 , piece_id=GKC5X2PJN2HZTRI6QXB6TTJMFJQE5JRKY4AY3USVH2NK4Z5GG6ZA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.422901513 (618h51m1.293984558s) [noise=true , address=45.91.116.12:4225 , piece_id=BWYR3KRDC65CEGYLKHWHLDKE5V7M4UNHZDS3JLBN5NWM4I26OVTQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.432415329 (618h51m1.303498374s) [noise=true , address=45.91.116.14:4192 , piece_id=AYN4RJZLNOMUAY4QSJNI7QAT27S5TRG34S7GXRGC7GOKWEMAPTPA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: <nil> Dec 31 10:20:01.432445657 (618h51m1.303528702s) [seg_pos={0 0}, upload=542312]: Segment reached optimal threshold of 13 pieces. Dec 31 10:20:01.432748964 (618h51m1.303832009s) [noise=true , address=45.91.116.8:4047 , piece_id=JT72P3DFK6KHUPXI6SRJLWQQJYKPDSBF3HFSAOW6K7SDSIF3PVHQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 31 10:20:01.432760426 (618h51m1.303843471s) [noise=true , address=45.91.116.8:4047 , piece_id=JT72P3DFK6KHUPXI6SRJLWQQJYKPDSBF3HFSAOW6K7SDSIF3PVHQ, seg_pos={0 0}, upload=542312]: Long tail cancellation due to optimal threshold Dec 31 10:20:01.433069074 (618h51m1.304152109s) [noise=true , address=45.91.116.10:4081 , piece_id=GK2ZYPBFP24JQZK24HX4DMNRB2Z55EODPO6XYFM52FQK7TBY4XUQ, seg_pos={0 0}, upload=542312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:59798->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:59798->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59798->45.91.116.10:4081: use of closed network connection Dec 31 10:20:01.433079604 (618h51m1.304162649s) [noise=true , address=45.91.116.10:4081 , piece_id=GK2ZYPBFP24JQZK24HX4DMNRB2Z55EODPO6XYFM52FQK7TBY4XUQ, seg_pos={0 0}, upload=542312]: Long tail cancellation due to optimal threshold Dec 31 10:20:01.433170958 (618h51m1.304254003s) [noise=true , address=45.91.116.13:4153 , piece_id=4NQ24M5CMKYN6SUAT7FMGBGDNOYI4Z6VY4B5AIRRYUS6MGSUBYRA, seg_pos={0 0}, upload=542312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 31 10:20:01.433218809 (618h51m1.304301854s) [noise=true , address=45.91.116.13:4153 , piece_id=4NQ24M5CMKYN6SUAT7FMGBGDNOYI4Z6VY4B5AIRRYUS6MGSUBYRA, seg_pos={0 0}, upload=542312]: Long tail cancellation due to optimal threshold Dec 31 10:20:01.433259757 (618h51m1.304342802s) [seg_pos={0 0}, upload=542312]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:20:01.433389514 (618h51m1.304472559s) [upload=542312]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:20:01.433398521 (618h51m1.304481566s) [upload=542312]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:20:01.726496276 (618h51m1.597579361s) [upload=542312]: Done uploading segments... Dec 31 10:20:01.726526283 (618h51m1.597609328s) [upload=542312]: Upload finished. err: <nil> Duration: 30.003579627s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.................X........X......X.........................] duration: 40.66943261s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 40.669450535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...XXX........XXX................XX......X....X.X...X.......] duration: 3.763576874s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:20:01 UTC) orbiter warning: context deadline exceeded Duration: 3.763603083s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....XXX........XXX................XX......X....X.X...X......] duration: 3.456373854s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:19:42 UTC) orbiter warning: context deadline exceeded Duration: 3.456403191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.................X........X......X........................] duration: 28.136245498s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 28.136273451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........X.....X....XXX..X.X.X....XX........X.XX..XX......X] duration: 30.003308288s trace: https://jaeger.dev.storj.io/trace/826d0457756b1e8 FAIL (2025-12-31 10:19:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 10:19:02.668824720 (618h50m2.539907775s) [upload=542301]: Starting upload Dec 31 10:19:02.668983041 (618h50m2.540066086s) [upload=542301]: Done starting upload Dec 31 10:19:02.669027395 (618h50m2.54011043s) [upload=542301]: Uploading segments... Dec 31 10:19:02.669316456 (618h50m2.540399522s) [upload=542301]: Got next segment. Inline: false Dec 31 10:19:02.669373314 (618h50m2.540456369s) [upload=542301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:19:02.669380107 (618h50m2.540463152s) [upload=542301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:19:02.918962742 (618h50m2.790045788s) [seg_pos={0 0}, upload=542301]: Begin upload segment... Dec 31 10:19:02.919036252 (618h50m2.790119297s) [seg_pos={0 0}, upload=542301]: Done begin upload segment. Dec 31 10:19:02.919100293 (618h50m2.790183339s) [noise=true , address=77.90.31.17:4770 , piece_id=ZIFXM5R6GSYSCG7TXFFFEQFPLN6WHORPNULUMTCDRO2JZWSED4XQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.919243798 (618h50m2.790326843s) [noise=true , address=145.79.227.19:4078 , piece_id=FJAGRQRTZ7L6KM37XDOXD6C6CGTJIN4GLBU4JC5R6BL4FHKCB4QA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.919909806 (618h50m2.790992851s) [noise=true , address=145.79.227.31:4798 , piece_id=TKMKUF65NFQOOTYZAW6ABHBF2U7IQTAZMOND3SEBL42P6OW6HODA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.920090069 (618h50m2.791173104s) [noise=true , address=145.79.227.40:4051 , piece_id=RMB6AQF4YKPKDCDJVAUYBKBBPRXT767KZ2ALJ2BQFHPOCC3J4FXQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.920461386 (618h50m2.791544421s) [noise=true , address=77.90.31.11:4555 , piece_id=NSE66OWBA3CBWSHBTZTPXXXNBASC2UYEL7Y4ENY6NPFMDK3WJMXA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.920573830 (618h50m2.791656875s) [noise=true , address=145.79.227.20:4438 , piece_id=ZUNUXPI6G25QGCOSTLHFA2PD4MBA4CUITHAOV5ETWN55JN34KDHA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921038885 (618h50m2.79212193s) [noise=true , address=145.79.227.17:4971 , piece_id=25H3ZYHRRZFBUPIF4B44AOQ5BMT7BLILSBESOB2IALXYAVXEC6OA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921239067 (618h50m2.792322132s) [noise=true , address=145.79.227.18:4007 , piece_id=DHHN7O5GGCOCUX4RGR4MT4R572K6N7CLE3DMOE5LNX7O5ISL3PKQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921482981 (618h50m2.792566026s) [noise=true , address=145.79.227.52:4447 , piece_id=OSYRG3C77FN3OEYFACAWA7GKE6NGEKIOI43F3I3SXLDDXWXMGAFA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921877191 (618h50m2.792960226s) [noise=true , address=145.79.227.41:4086 , piece_id=G76WDVBUHMRB2MPGN6XLFSX7UUU4FGNP6SUN5RMSEOKL6WRE2LJQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.921922017 (618h50m2.793005053s) [noise=true , address=77.90.31.18:4805 , piece_id=6RCLHIZDZDA74BSEB3PJC3VDF4MDLV5NSN24LAVF2OWPIQS4XDAQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.922412351 (618h50m2.793495396s) [noise=true , address=77.90.31.16:4736 , piece_id=SYH636EFJAWJM327IVXI25TEPC2CUBZ3DMHUS2PALLTGEMFZJEVA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.923331862 (618h50m2.794414907s) [noise=true , address=45.91.116.9:4262 , piece_id=JANELRJQ4ELLBM7F5E7IHCZBGZFRGQHGZZNT3VACSCAFRN42KBRA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.924332477 (618h50m2.795415522s) [noise=true , address=77.90.31.14:4661 , piece_id=4N3QF2SV4SXYPCVRKDUU5MIWNB5XCQLJ2JDI6YAXMX33KULHRVVA, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.925226188 (618h50m2.796309233s) [noise=true , address=77.90.31.15:4700 , piece_id=S5OBGPYVRLHBVZH6O6R2NTR2QOXC6RK67CJ6JUGIMAF2CYO2AHLQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:02.926041160 (618h50m2.797124205s) [noise=true , address=145.79.227.7:4293 , piece_id=2744XVEUVN5DAMD3RV4Q6FT7X6WM3SBHX3SZV2EA3BPDRDHNBPRQ, seg_pos={0 0}, upload=542301]: Uploading piece... Dec 31 10:19:04.621137329 (618h50m4.492220364s) [upload=542301]: Next returned nil segment Dec 31 10:19:04.621163348 (618h50m4.492246393s) [upload=542301]: Waiting for error group managing segments... Dec 31 10:19:04.905177926 (618h50m4.776260961s) [noise=true , address=145.79.227.31:4798 , piece_id=TKMKUF65NFQOOTYZAW6ABHBF2U7IQTAZMOND3SEBL42P6OW6HODA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.909692923 (618h50m4.780775958s) [noise=true , address=145.79.227.20:4438 , piece_id=ZUNUXPI6G25QGCOSTLHFA2PD4MBA4CUITHAOV5ETWN55JN34KDHA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.918998833 (618h50m4.790081878s) [noise=true , address=145.79.227.19:4078 , piece_id=FJAGRQRTZ7L6KM37XDOXD6C6CGTJIN4GLBU4JC5R6BL4FHKCB4QA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.932095681 (618h50m4.803178716s) [noise=true , address=145.79.227.41:4086 , piece_id=G76WDVBUHMRB2MPGN6XLFSX7UUU4FGNP6SUN5RMSEOKL6WRE2LJQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.934577077 (618h50m4.805660112s) [noise=true , address=145.79.227.7:4293 , piece_id=2744XVEUVN5DAMD3RV4Q6FT7X6WM3SBHX3SZV2EA3BPDRDHNBPRQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.936571775 (618h50m4.80765482s) [noise=true , address=145.79.227.52:4447 , piece_id=OSYRG3C77FN3OEYFACAWA7GKE6NGEKIOI43F3I3SXLDDXWXMGAFA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:04.989521850 (618h50m4.860604885s) [noise=true , address=145.79.227.17:4971 , piece_id=25H3ZYHRRZFBUPIF4B44AOQ5BMT7BLILSBESOB2IALXYAVXEC6OA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.078679741 (618h50m4.949762786s) [noise=true , address=145.79.227.18:4007 , piece_id=DHHN7O5GGCOCUX4RGR4MT4R572K6N7CLE3DMOE5LNX7O5ISL3PKQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.159365899 (618h50m5.030448944s) [noise=true , address=77.90.31.17:4770 , piece_id=ZIFXM5R6GSYSCG7TXFFFEQFPLN6WHORPNULUMTCDRO2JZWSED4XQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.209984214 (618h50m5.081067259s) [noise=true , address=77.90.31.11:4555 , piece_id=NSE66OWBA3CBWSHBTZTPXXXNBASC2UYEL7Y4ENY6NPFMDK3WJMXA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.340698936 (618h50m5.211781971s) [noise=true , address=45.91.116.9:4262 , piece_id=JANELRJQ4ELLBM7F5E7IHCZBGZFRGQHGZZNT3VACSCAFRN42KBRA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.753148975 (618h50m5.624232021s) [noise=true , address=77.90.31.16:4736 , piece_id=SYH636EFJAWJM327IVXI25TEPC2CUBZ3DMHUS2PALLTGEMFZJEVA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.753904654 (618h50m5.624987699s) [noise=true , address=77.90.31.14:4661 , piece_id=4N3QF2SV4SXYPCVRKDUU5MIWNB5XCQLJ2JDI6YAXMX33KULHRVVA, seg_pos={0 0}, upload=542301]: Done uploading piece. err: <nil> Dec 31 10:19:05.753917409 (618h50m5.625000444s) [seg_pos={0 0}, upload=542301]: Segment reached optimal threshold of 13 pieces. Dec 31 10:19:05.754157135 (618h50m5.62524017s) [noise=true , address=77.90.31.15:4700 , piece_id=S5OBGPYVRLHBVZH6O6R2NTR2QOXC6RK67CJ6JUGIMAF2CYO2AHLQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 31 10:19:05.754166132 (618h50m5.625249177s) [noise=true , address=77.90.31.15:4700 , piece_id=S5OBGPYVRLHBVZH6O6R2NTR2QOXC6RK67CJ6JUGIMAF2CYO2AHLQ, seg_pos={0 0}, upload=542301]: Long tail cancellation due to optimal threshold Dec 31 10:19:05.754172955 (618h50m5.625256s) [noise=true , address=77.90.31.18:4805 , piece_id=6RCLHIZDZDA74BSEB3PJC3VDF4MDLV5NSN24LAVF2OWPIQS4XDAQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 31 10:19:05.754183394 (618h50m5.625266429s) [noise=true , address=77.90.31.18:4805 , piece_id=6RCLHIZDZDA74BSEB3PJC3VDF4MDLV5NSN24LAVF2OWPIQS4XDAQ, seg_pos={0 0}, upload=542301]: Long tail cancellation due to optimal threshold Dec 31 10:19:05.754238329 (618h50m5.625321375s) [noise=true , address=145.79.227.40:4051 , piece_id=RMB6AQF4YKPKDCDJVAUYBKBBPRXT767KZ2ALJ2BQFHPOCC3J4FXQ, seg_pos={0 0}, upload=542301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 10:19:05.754243219 (618h50m5.625326264s) [noise=true , address=145.79.227.40:4051 , piece_id=RMB6AQF4YKPKDCDJVAUYBKBBPRXT767KZ2ALJ2BQFHPOCC3J4FXQ, seg_pos={0 0}, upload=542301]: Long tail cancellation due to optimal threshold Dec 31 10:19:05.754258298 (618h50m5.625341343s) [seg_pos={0 0}, upload=542301]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:19:05.754339061 (618h50m5.625422106s) [upload=542301]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:19:05.754345774 (618h50m5.625428809s) [upload=542301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:19:06.054101826 (618h50m5.925184871s) [upload=542301]: Done uploading segments... Dec 31 10:19:06.054126382 (618h50m5.925209427s) [upload=542301]: Upload finished. err: <nil> Duration: 30.003335971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....XXX........XXX................XX......X....X.X...X.....] duration: 3.569440472s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:18:47 UTC) orbiter warning: context deadline exceeded Duration: 3.569466712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.................X........X......X.......................] duration: 19.779008294s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 19.7790372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.................X........X......X......................] duration: 32.515594833s trace: https://jaeger.dev.storj.io/trace/5eeb6c8ca9bb2595 FAIL (2025-12-31 10:18:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== :10.873846354 (618h30m10.744929399s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874394749 (618h30m10.745477784s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.874764903 (618h30m10.745847938s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875179162 (618h30m10.746262197s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.875700835 (618h30m10.74678388s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876211989 (618h30m10.747295024s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876727740 (618h30m10.747810785s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.876928623 (618h30m10.748011668s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877707706 (618h30m10.748790751s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.877790354 (618h30m10.748873399s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.878880610 (618h30m10.749963655s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879641619 (618h30m10.750724664s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.879826230 (618h30m10.750909275s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.881143238 (618h30m10.752226333s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.882905023 (618h30m10.753988088s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.888135012 (618h30m10.759218057s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:10.892487259 (618h30m10.763570304s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Uploading piece... Dec 31 09:59:12.306596361 (618h30m12.177679396s) [upload=542009]: Next returned nil segment Dec 31 09:59:12.306609827 (618h30m12.177692862s) [upload=542009]: Waiting for error group managing segments... Dec 31 09:59:12.757377273 (618h30m12.628460308s) [noise=true , address=145.79.227.16:4929 , piece_id=KEV2FI5IS3X2IJUUDWC7VNXYE77B22ZWAYPXO3QHIAPMO6AZUQTQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772634234 (618h30m12.643717269s) [noise=true , address=145.79.227.18:4031 , piece_id=3TH3TDI4G5JI2KM733E54MFXRZBNCOFPRW7XSX5VEW4ZRUUC463A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.772690512 (618h30m12.643773547s) [noise=true , address=145.79.227.6:4271 , piece_id=MUYUCBVFIFUUKNTOYDPPOOVHK62GBZLMPLOYOFCKYZVQRUFE3FDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.779721769 (618h30m12.650804814s) [noise=true , address=64.49.13.170:4720 , piece_id=6MOX5BFK7L3L3IN73YRZ2KMDY3DSD7E5KP7OQGHYBPNEGWYVAOXA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.807574997 (618h30m12.678658042s) [noise=true , address=145.79.227.5:4239 , piece_id=U6EKF73EGHOLFI2KXN3ORNSQSEULLTAMR22FZ5WV3EVXQUCAST6A, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.915126329 (618h30m12.786209364s) [noise=true , address=77.90.31.13:4640 , piece_id=HCDFLAFUROZ5BX5BVWTW6BSBYVLBI63B2ILQTZ47J6ZGAHJFNMDQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.917160994 (618h30m12.788244029s) [noise=true , address=77.90.31.18:4831 , piece_id=4P3FQBRDI5D2ANYXPWFTUGRTQIEPIHLAMZEM574XM44ELNHSBANQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:12.931944514 (618h30m12.803027549s) [noise=true , address=77.90.31.19:4857 , piece_id=N7XT4NIOSZ5FY6FIIWDFG2F7G4SSMR32Q3BR5VXUYM6HMGYPKWQA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.140715014 (618h30m13.011798059s) [noise=true , address=45.91.116.11:4144 , piece_id=CD2QYYKADWDNQ4PX6WAWZRIJ4CAB2YGTT2X4RXDNHYDBP4MSFBKQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.161757762 (618h30m13.032840797s) [noise=true , address=45.91.116.7:4013 , piece_id=GAX4JZPYTANMSTZGDOL7UV27DCYYLNDPTPJIFRS5UYXE6OVDDEHA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.623650675 (618h30m13.49473371s) [noise=true , address=145.79.227.30:4769 , piece_id=QQWQ7GO7TWPFRDRZQE5AIWAV52KNEGHTBMAQ3H7RDVQJ46AZIDCA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.633548713 (618h30m13.504631758s) [noise=true , address=145.79.227.44:4223 , piece_id=3ED6D4CDTBVI3DF24XINEG4V3V24ETRB7WITGRCZJJLQ5TB2UGGQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.664204926 (618h30m13.535287961s) [noise=true , address=145.79.227.40:4053 , piece_id=YVMVVHIMXWCVMIOFO5CAPNDRODREWCOTPQTWXITCXHYFGUVPVMDA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704160307 (618h30m13.575243342s) [noise=true , address=145.79.227.49:4340 , piece_id=24LLNGQZN7AH7M7XDSNV2EEJAEJ4TQZJETC5YQJX7HW3SWBAVONA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: <nil> Dec 31 09:59:13.704180917 (618h30m13.575263962s) [seg_pos={0 0}, upload=542009]: Segment reached optimal threshold of 14 pieces. Dec 31 09:59:13.704371108 (618h30m13.575454153s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704380175 (618h30m13.57546321s) [noise=true , address=145.79.227.43:4189 , piece_id=NSNZQHVPJQF76IMRRWTDIVCIOKF4HPIHLKD6AYJHLTSFU5OWPLTA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704497289 (618h30m13.575580324s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1c6o4CCFBx5xgnmaivGFzRGqqp8qrqwyzwTrcpwDYFh71wRQSZ): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704502889 (618h30m13.575585924s) [noise=true , address=145.79.227.35:4510 , piece_id=S2O37JE6YW4PLR4Z6KJQYTZN7F74FJFMUYL4EYW2H5UGLX7SBXIA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704573113 (618h30m13.575656148s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704589153 (618h30m13.575672198s) [noise=true , address=193.149.164.143:4704 , piece_id=4MXPCIEJBFGKSLIN2JRFQB5KGXJPGOEVXDWRB723X7JTS2ZVFMQQ, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704601768 (618h30m13.575684813s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 31 09:59:13.704609292 (618h30m13.575692327s) [noise=true , address=145.79.227.34:4919 , piece_id=VS3JGFO47VC3QYGUSZZIOT63YAQXDWIZ4DHNE4MBB2D6CF3CU2MA, seg_pos={0 0}, upload=542009]: Long tail cancellation due to optimal threshold Dec 31 09:59:13.704651793 (618h30m13.575734838s) [seg_pos={0 0}, upload=542009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 31 09:59:13.704730402 (618h30m13.575813437s) [upload=542009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 09:59:13.704735633 (618h30m13.575818678s) [upload=542009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 09:59:13.997689169 (618h30m13.868772204s) [upload=542009]: Done uploading segments... Dec 31 09:59:13.997709939 (618h30m13.868792974s) [upload=542009]: Upload finished. err: <nil> Duration: 32.515621935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......XXX........XXX................XX......X....X.X...X....] duration: 3.678475329s trace: https://jaeger.dev.storj.io/trace/66892903c3efdb71 FAIL (2025-12-31 10:18:20 UTC) orbiter warning: context deadline exceeded Duration: 3.678503974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..........X.....X....XXX..X.X.X....XX........X.XX..XX......] duration: 5.959309764s trace: https://jaeger.dev.storj.io/trace/34c911959fb77d76 FAIL (2025-12-31 10:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA) context deadline exceeded ======LOGS====== :14:27.797368579 (618h45m27.668451614s) [upload=542232]: Starting upload Dec 31 10:14:27.797422823 (618h45m27.668505858s) [upload=542232]: Done starting upload Dec 31 10:14:27.797548511 (618h45m27.668631546s) [upload=542232]: Uploading segments... Dec 31 10:14:27.797602364 (618h45m27.668685409s) [upload=542232]: Got next segment. Inline: false Dec 31 10:14:27.797644384 (618h45m27.668727429s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 10:14:27.797649514 (618h45m27.668732559s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 10:14:28.040762048 (618h45m27.911845083s) [seg_pos={0 0}, upload=542232]: Begin upload segment... Dec 31 10:14:28.040802645 (618h45m27.91188568s) [seg_pos={0 0}, upload=542232]: Done begin upload segment. Dec 31 10:14:28.040827944 (618h45m27.911910989s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.040863331 (618h45m27.911946366s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041342332 (618h45m27.912425367s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041636804 (618h45m27.912719839s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.041946053 (618h45m27.913029088s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042322550 (618h45m27.913405585s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042328291 (618h45m27.913411326s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042622160 (618h45m27.913705205s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042683036 (618h45m27.913766071s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.042906250 (618h45m27.913989286s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043086434 (618h45m27.914169469s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.043187516 (618h45m27.914270552s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.044624794 (618h45m27.915707829s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045016018 (618h45m27.916099053s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045498066 (618h45m27.916581101s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:28.045573049 (618h45m27.916656084s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Uploading piece... Dec 31 10:14:29.735961201 (618h45m29.607044236s) [upload=542232]: Next returned nil segment Dec 31 10:14:29.735974056 (618h45m29.607057091s) [upload=542232]: Waiting for error group managing segments... Dec 31 10:14:30.022384897 (618h45m29.893467932s) [noise=true , address=145.79.227.36:4519 , piece_id=BKBT66VLNGJPVUFES54BVCLL3JX6S6RJ5PHOH6OFQWVXZKVGJYKQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.034018719 (618h45m29.905101754s) [noise=true , address=145.79.227.51:4409 , piece_id=NG4UD3OKKOWKPHW2TXLHF2CEX33S335XIJSJNFBIPAWUHUZI2DVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.042007260 (618h45m29.913090305s) [noise=true , address=145.79.227.18:4006 , piece_id=5GIY655XCBAVXXKBLY6WB3OKHUABB764TYHDJP7C3ZMX3KFW4HQQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050266508 (618h45m29.921349553s) [noise=true , address=145.79.227.32:4834 , piece_id=FTS4RPS4GQLG5PH6U4XBA4NI7ISH54KA2YTHQRQKCXWFVDYNVBBA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.050303619 (618h45m29.921386654s) [noise=true , address=145.79.227.38:4978 , piece_id=XRTOHRFRTJAZJW64ZJK5ISKYKXCWFALFCVQ4N3JA56BHSIFWSDVA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.052857992 (618h45m29.923941027s) [noise=true , address=145.79.227.10:4401 , piece_id=ILETMLA4XR7OP645QKREM5SLUEJOKYJU325P7LDWLLLQ7KO5WSNQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100372507 (618h45m29.971455552s) [noise=true , address=145.79.227.3:4150 , piece_id=WFNJZQJBVBMHXV5ULNBIXFURZKK4BMGRHZMRHBQS4OQIRO36AAHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.100665835 (618h45m29.971748871s) [noise=true , address=145.79.227.8:4329 , piece_id=YUSSW5VP3ECKVKSU4A5RZBEAWGESKWXE7U5NTTIPWSCQ7FDIXSOA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.118643639 (618h45m29.989726674s) [noise=true , address=145.79.227.27:4689 , piece_id=THKA733ZHBJDOBLODFYVKYNCOZMYL4ABG6CTK3XM5GMTLHQ5IMYQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.273428402 (618h45m30.144511447s) [noise=true , address=77.90.31.12:4589 , piece_id=KHITMFYXG5GRYOLFKKH327REZMTBENOQBIGRGHFYOUWJCXRAOPHA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.326746749 (618h45m30.197829784s) [noise=true , address=77.90.31.14:4663 , piece_id=RYGEL7SGWLIWFQD7TXPXTX77GG5MFWJVOCCXOPRC4SYM3X2Q5G5A, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.338978621 (618h45m30.210061656s) [noise=true , address=45.91.116.13:4154 , piece_id=KHOEELOHFFHQJYUCD2JLRYUHMMMSZCYBMR2Y6BTBCNX44UOD4PJQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344144688 (618h45m30.215227723s) [noise=true , address=45.91.116.4:4937 , piece_id=DJZI5D2UNI3W4VCSTPTZGUT44ISLFKPENL56T5T7AZSPRVMGTQAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: <nil> Dec 31 10:14:30.344170737 (618h45m30.215253782s) [seg_pos={0 0}, upload=542232]: Segment reached optimal threshold of 13 pieces. Dec 31 10:14:30.344382710 (618h45m30.215465755s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48176->45.91.116.12:4226: use of closed network connection Dec 31 10:14:30.344391407 (618h45m30.215474452s) [noise=true , address=45.91.116.12:4226 , piece_id=FA5CTC5LNQF62DVLSVTYNZ2CQRUC2IGIEF6OPOT2LFCBXGVJMMEQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.344943459 (618h45m30.216026494s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.344956433 (618h45m30.216039478s) [noise=true , address=45.91.116.14:4191 , piece_id=YUAPZ4LMYSP3XEVG6IXDZ6OVXBHE66QERVVG75DHHNUD26JVJDAA, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345267235 (618h45m30.21635028s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 31 10:14:30.345274298 (618h45m30.216357333s) [noise=true , address=45.91.116.7:4011 , piece_id=KGOLNHNDSW5URXRHRURGE6TD5NYMFJVHLBKPVUH5TF7GLAI6KBOQ, seg_pos={0 0}, upload=542232]: Long tail cancellation due to optimal threshold Dec 31 10:14:30.345291902 (618h45m30.216374937s) [seg_pos={0 0}, upload=542232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 10:14:30.345462737 (618h45m30.216545772s) [upload=542232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 10:14:30.345469180 (618h45m30.216552215s) [upload=542232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 10:14:30.637355706 (618h45m30.508438741s) [upload=542232]: Done uploading segments... Dec 31 10:14:30.637373821 (618h45m30.508456856s) [upload=542232]: Upload finished. err: <nil> Duration: 5.959338268s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)