Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 901134 reports (Page 11324 of 18023)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 16.999276641s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:22:08 UTC) orbiter warning: context deadline exceeded Duration: 16.999320295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XXX.X.XX] duration: 5.001245155s trace: https://jaeger.dev.storj.io/trace/51387dcffc6cfac FAIL (2025-12-28 16:21:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00126897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 18.419670334s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:21:43 UTC) orbiter warning: context deadline exceeded Duration: 18.420741713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 12.210718548s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:21:21 UTC) orbiter warning: context deadline exceeded Duration: 12.210768263s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 18.220563192s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:21:11 UTC) orbiter warning: context deadline exceeded Duration: 18.220579574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X........X.X.X.X....XXX...X...X.......X....XXXXX.........X..] duration: 5.88385274s trace: https://jaeger.dev.storj.io/trace/6221cb65a29c769b FAIL (2025-12-28 16:21:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 16:19:41.477959304 (552h50m41.349042339s) [upload=484702]: Starting upload Dec 28 16:19:41.478023667 (552h50m41.349106702s) [upload=484702]: Done starting upload Dec 28 16:19:41.478151841 (552h50m41.349234876s) [upload=484702]: Uploading segments... Dec 28 16:19:41.478456340 (552h50m41.349539385s) [upload=484702]: Got next segment. Inline: false Dec 28 16:19:41.478490636 (552h50m41.349573671s) [upload=484702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:19:41.478496388 (552h50m41.349579433s) [upload=484702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:19:41.725457381 (552h50m41.596540417s) [seg_pos={0 0}, upload=484702]: Begin upload segment... Dec 28 16:19:41.725496486 (552h50m41.596579531s) [seg_pos={0 0}, upload=484702]: Done begin upload segment. Dec 28 16:19:41.725524059 (552h50m41.596607094s) [noise=true , address=45.91.116.6:4976 , piece_id=ZTY24CLXXPKEZFYCSVLEEWEWCDYSCW7NCSV2NUP46Q2T7V2GFGBQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.725551511 (552h50m41.596634556s) [noise=true , address=145.79.227.16:4912 , piece_id=7FG7MRV7SP7CBDJXG6GSYMALXDB35RHGUVIC4Y6OCFS2MK6NV3LA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.726206979 (552h50m41.597290024s) [noise=true , address=145.79.227.15:4874 , piece_id=LUB5T7WNOEDSHUROEK2NTBFGY37P4JYAJHVN3PPMIHZ3D2LLX37Q, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.727712476 (552h50m41.598795511s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728009310 (552h50m41.599092356s) [noise=true , address=145.79.227.47:4265 , piece_id=5FWNDQJA7CWQTDO2R3RW3APWKKAOYVXJL4MIIWDKLPE2NHTOFOAQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728304032 (552h50m41.599387067s) [noise=true , address=45.91.116.7:4009 , piece_id=TUOAYAJ5T54RZWFKRKMK3M77TP4BG2BZ5ZAXE3FDIF2Q4TAN6L2Q, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728621737 (552h50m41.599704782s) [noise=true , address=45.91.116.13:4153 , piece_id=2YNDCGVX5NF2GOKEI4FL4YVI6HR2CEPPPJBRJV3VZEURTFXR2ZMA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728939181 (552h50m41.600022216s) [noise=true , address=45.91.116.14:4189 , piece_id=BTSWH5ZRVXMWSK6GNZC4ZBIALJYCRBXAOPMNAQSYPTHPBQQVHMSQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729068398 (552h50m41.600151443s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729228292 (552h50m41.600311327s) [noise=true , address=145.79.227.38:4977 , piece_id=LW6NZHQINWAZ4BLHYTBWBUL7NCSYXBMPC7SBO5BRW5DKGTOZOIJQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729455825 (552h50m41.60053887s) [noise=true , address=145.79.227.46:4603 , piece_id=O552YHMQL2DYQLS2PEMZTHCCU3SYSCNOKVZOCRIDJ5K2Z75JPEVA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729493387 (552h50m41.600576432s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729870485 (552h50m41.60095352s) [noise=true , address=145.79.227.14:4838 , piece_id=NGHELVUBZ6KO2IMLG64BPVNLCMKROZAFFJ4OWE6C6PDNM4QRWRLQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.730441833 (552h50m41.601524878s) [noise=true , address=145.79.227.7:4294 , piece_id=N6KQ6PKWQ7OEORBA3TZAPTTDGEKPMLSFELAJHHLBPDF2OZMRY7NQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.730926756 (552h50m41.602009801s) [noise=true , address=145.79.227.9:4366 , piece_id=OIF7RU62XF5XHBP4M64GU2OBYRRAOLXAHCYXGT54O726OJDMFN2A, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.731333130 (552h50m41.602416165s) [noise=true , address=145.79.227.48:4303 , piece_id=FDJYNMU57R6XF4QUK32KFNIQX5TM3B6SSZKFDTFXR6J6BE6E6KIQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:43.439258955 (552h50m43.310342s) [upload=484702]: Next returned nil segment Dec 28 16:19:43.439502248 (552h50m43.310585323s) [upload=484702]: Waiting for error group managing segments... Dec 28 16:19:43.726760847 (552h50m43.597843892s) [noise=true , address=145.79.227.9:4366 , piece_id=OIF7RU62XF5XHBP4M64GU2OBYRRAOLXAHCYXGT54O726OJDMFN2A, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.726797166 (552h50m43.597880211s) [noise=true , address=145.79.227.7:4294 , piece_id=N6KQ6PKWQ7OEORBA3TZAPTTDGEKPMLSFELAJHHLBPDF2OZMRY7NQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.734570077 (552h50m43.605653112s) [noise=true , address=145.79.227.46:4603 , piece_id=O552YHMQL2DYQLS2PEMZTHCCU3SYSCNOKVZOCRIDJ5K2Z75JPEVA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.738146557 (552h50m43.609229602s) [noise=true , address=145.79.227.48:4303 , piece_id=FDJYNMU57R6XF4QUK32KFNIQX5TM3B6SSZKFDTFXR6J6BE6E6KIQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.748516273 (552h50m43.619599358s) [noise=true , address=145.79.227.38:4977 , piece_id=LW6NZHQINWAZ4BLHYTBWBUL7NCSYXBMPC7SBO5BRW5DKGTOZOIJQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.752687726 (552h50m43.623770791s) [noise=true , address=145.79.227.47:4265 , piece_id=5FWNDQJA7CWQTDO2R3RW3APWKKAOYVXJL4MIIWDKLPE2NHTOFOAQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.756612580 (552h50m43.627695625s) [noise=true , address=145.79.227.15:4874 , piece_id=LUB5T7WNOEDSHUROEK2NTBFGY37P4JYAJHVN3PPMIHZ3D2LLX37Q, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.763676310 (552h50m43.634759355s) [noise=true , address=145.79.227.14:4838 , piece_id=NGHELVUBZ6KO2IMLG64BPVNLCMKROZAFFJ4OWE6C6PDNM4QRWRLQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.768023057 (552h50m43.639106092s) [noise=true , address=145.79.227.16:4912 , piece_id=7FG7MRV7SP7CBDJXG6GSYMALXDB35RHGUVIC4Y6OCFS2MK6NV3LA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.083729315 (552h50m43.95481235s) [noise=true , address=45.91.116.13:4153 , piece_id=2YNDCGVX5NF2GOKEI4FL4YVI6HR2CEPPPJBRJV3VZEURTFXR2ZMA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.084971881 (552h50m43.956054926s) [noise=true , address=45.91.116.7:4009 , piece_id=TUOAYAJ5T54RZWFKRKMK3M77TP4BG2BZ5ZAXE3FDIF2Q4TAN6L2Q, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.163872059 (552h50m44.034955104s) [noise=true , address=45.91.116.14:4189 , piece_id=BTSWH5ZRVXMWSK6GNZC4ZBIALJYCRBXAOPMNAQSYPTHPBQQVHMSQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.666546576 (552h50m44.537629611s) [noise=true , address=45.91.116.6:4976 , piece_id=ZTY24CLXXPKEZFYCSVLEEWEWCDYSCW7NCSV2NUP46Q2T7V2GFGBQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.666569931 (552h50m44.537652966s) [seg_pos={0 0}, upload=484702]: Segment reached optimal threshold of 13 pieces. Dec 28 16:19:44.666870052 (552h50m44.537953107s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 16:19:44.666879230 (552h50m44.537962285s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667143703 (552h50m44.538226758s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 16:19:44.667162900 (552h50m44.538245935s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667235388 (552h50m44.538318433s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection Dec 28 16:19:44.667245407 (552h50m44.538328452s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667267780 (552h50m44.538350815s) [seg_pos={0 0}, upload=484702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:19:44.667361729 (552h50m44.538444764s) [upload=484702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:19:44.667367830 (552h50m44.538450865s) [upload=484702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:19:45.021706490 (552h50m44.892789525s) [upload=484702]: Done uploading segments... Dec 28 16:19:45.021725065 (552h50m44.89280811s) [upload=484702]: Upload finished. err: <nil> Duration: 5.883885172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....................X.............X..........] duration: 23.719948441s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:20:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 23.719964111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XXX.X.X] duration: 5.00062689s trace: https://jaeger.dev.storj.io/trace/4edf6f1bab159d6e FAIL (2025-12-28 16:20:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000766697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X........X.X.X.X....XXX...X...X.......X....XXXXX.........X.] duration: 6.843182765s trace: https://jaeger.dev.storj.io/trace/6221cb65a29c769b FAIL (2025-12-28 16:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 16:19:41.477959304 (552h50m41.349042339s) [upload=484702]: Starting upload Dec 28 16:19:41.478023667 (552h50m41.349106702s) [upload=484702]: Done starting upload Dec 28 16:19:41.478151841 (552h50m41.349234876s) [upload=484702]: Uploading segments... Dec 28 16:19:41.478456340 (552h50m41.349539385s) [upload=484702]: Got next segment. Inline: false Dec 28 16:19:41.478490636 (552h50m41.349573671s) [upload=484702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:19:41.478496388 (552h50m41.349579433s) [upload=484702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:19:41.725457381 (552h50m41.596540417s) [seg_pos={0 0}, upload=484702]: Begin upload segment... Dec 28 16:19:41.725496486 (552h50m41.596579531s) [seg_pos={0 0}, upload=484702]: Done begin upload segment. Dec 28 16:19:41.725524059 (552h50m41.596607094s) [noise=true , address=45.91.116.6:4976 , piece_id=ZTY24CLXXPKEZFYCSVLEEWEWCDYSCW7NCSV2NUP46Q2T7V2GFGBQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.725551511 (552h50m41.596634556s) [noise=true , address=145.79.227.16:4912 , piece_id=7FG7MRV7SP7CBDJXG6GSYMALXDB35RHGUVIC4Y6OCFS2MK6NV3LA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.726206979 (552h50m41.597290024s) [noise=true , address=145.79.227.15:4874 , piece_id=LUB5T7WNOEDSHUROEK2NTBFGY37P4JYAJHVN3PPMIHZ3D2LLX37Q, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.727712476 (552h50m41.598795511s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728009310 (552h50m41.599092356s) [noise=true , address=145.79.227.47:4265 , piece_id=5FWNDQJA7CWQTDO2R3RW3APWKKAOYVXJL4MIIWDKLPE2NHTOFOAQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728304032 (552h50m41.599387067s) [noise=true , address=45.91.116.7:4009 , piece_id=TUOAYAJ5T54RZWFKRKMK3M77TP4BG2BZ5ZAXE3FDIF2Q4TAN6L2Q, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728621737 (552h50m41.599704782s) [noise=true , address=45.91.116.13:4153 , piece_id=2YNDCGVX5NF2GOKEI4FL4YVI6HR2CEPPPJBRJV3VZEURTFXR2ZMA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728939181 (552h50m41.600022216s) [noise=true , address=45.91.116.14:4189 , piece_id=BTSWH5ZRVXMWSK6GNZC4ZBIALJYCRBXAOPMNAQSYPTHPBQQVHMSQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729068398 (552h50m41.600151443s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729228292 (552h50m41.600311327s) [noise=true , address=145.79.227.38:4977 , piece_id=LW6NZHQINWAZ4BLHYTBWBUL7NCSYXBMPC7SBO5BRW5DKGTOZOIJQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729455825 (552h50m41.60053887s) [noise=true , address=145.79.227.46:4603 , piece_id=O552YHMQL2DYQLS2PEMZTHCCU3SYSCNOKVZOCRIDJ5K2Z75JPEVA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729493387 (552h50m41.600576432s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729870485 (552h50m41.60095352s) [noise=true , address=145.79.227.14:4838 , piece_id=NGHELVUBZ6KO2IMLG64BPVNLCMKROZAFFJ4OWE6C6PDNM4QRWRLQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.730441833 (552h50m41.601524878s) [noise=true , address=145.79.227.7:4294 , piece_id=N6KQ6PKWQ7OEORBA3TZAPTTDGEKPMLSFELAJHHLBPDF2OZMRY7NQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.730926756 (552h50m41.602009801s) [noise=true , address=145.79.227.9:4366 , piece_id=OIF7RU62XF5XHBP4M64GU2OBYRRAOLXAHCYXGT54O726OJDMFN2A, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.731333130 (552h50m41.602416165s) [noise=true , address=145.79.227.48:4303 , piece_id=FDJYNMU57R6XF4QUK32KFNIQX5TM3B6SSZKFDTFXR6J6BE6E6KIQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:43.439258955 (552h50m43.310342s) [upload=484702]: Next returned nil segment Dec 28 16:19:43.439502248 (552h50m43.310585323s) [upload=484702]: Waiting for error group managing segments... Dec 28 16:19:43.726760847 (552h50m43.597843892s) [noise=true , address=145.79.227.9:4366 , piece_id=OIF7RU62XF5XHBP4M64GU2OBYRRAOLXAHCYXGT54O726OJDMFN2A, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.726797166 (552h50m43.597880211s) [noise=true , address=145.79.227.7:4294 , piece_id=N6KQ6PKWQ7OEORBA3TZAPTTDGEKPMLSFELAJHHLBPDF2OZMRY7NQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.734570077 (552h50m43.605653112s) [noise=true , address=145.79.227.46:4603 , piece_id=O552YHMQL2DYQLS2PEMZTHCCU3SYSCNOKVZOCRIDJ5K2Z75JPEVA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.738146557 (552h50m43.609229602s) [noise=true , address=145.79.227.48:4303 , piece_id=FDJYNMU57R6XF4QUK32KFNIQX5TM3B6SSZKFDTFXR6J6BE6E6KIQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.748516273 (552h50m43.619599358s) [noise=true , address=145.79.227.38:4977 , piece_id=LW6NZHQINWAZ4BLHYTBWBUL7NCSYXBMPC7SBO5BRW5DKGTOZOIJQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.752687726 (552h50m43.623770791s) [noise=true , address=145.79.227.47:4265 , piece_id=5FWNDQJA7CWQTDO2R3RW3APWKKAOYVXJL4MIIWDKLPE2NHTOFOAQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.756612580 (552h50m43.627695625s) [noise=true , address=145.79.227.15:4874 , piece_id=LUB5T7WNOEDSHUROEK2NTBFGY37P4JYAJHVN3PPMIHZ3D2LLX37Q, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.763676310 (552h50m43.634759355s) [noise=true , address=145.79.227.14:4838 , piece_id=NGHELVUBZ6KO2IMLG64BPVNLCMKROZAFFJ4OWE6C6PDNM4QRWRLQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.768023057 (552h50m43.639106092s) [noise=true , address=145.79.227.16:4912 , piece_id=7FG7MRV7SP7CBDJXG6GSYMALXDB35RHGUVIC4Y6OCFS2MK6NV3LA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.083729315 (552h50m43.95481235s) [noise=true , address=45.91.116.13:4153 , piece_id=2YNDCGVX5NF2GOKEI4FL4YVI6HR2CEPPPJBRJV3VZEURTFXR2ZMA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.084971881 (552h50m43.956054926s) [noise=true , address=45.91.116.7:4009 , piece_id=TUOAYAJ5T54RZWFKRKMK3M77TP4BG2BZ5ZAXE3FDIF2Q4TAN6L2Q, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.163872059 (552h50m44.034955104s) [noise=true , address=45.91.116.14:4189 , piece_id=BTSWH5ZRVXMWSK6GNZC4ZBIALJYCRBXAOPMNAQSYPTHPBQQVHMSQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.666546576 (552h50m44.537629611s) [noise=true , address=45.91.116.6:4976 , piece_id=ZTY24CLXXPKEZFYCSVLEEWEWCDYSCW7NCSV2NUP46Q2T7V2GFGBQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.666569931 (552h50m44.537652966s) [seg_pos={0 0}, upload=484702]: Segment reached optimal threshold of 13 pieces. Dec 28 16:19:44.666870052 (552h50m44.537953107s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 16:19:44.666879230 (552h50m44.537962285s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667143703 (552h50m44.538226758s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 16:19:44.667162900 (552h50m44.538245935s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667235388 (552h50m44.538318433s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection Dec 28 16:19:44.667245407 (552h50m44.538328452s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667267780 (552h50m44.538350815s) [seg_pos={0 0}, upload=484702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:19:44.667361729 (552h50m44.538444764s) [upload=484702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:19:44.667367830 (552h50m44.538450865s) [upload=484702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:19:45.021706490 (552h50m44.892789525s) [upload=484702]: Done uploading segments... Dec 28 16:19:45.021725065 (552h50m44.89280811s) [upload=484702]: Upload finished. err: <nil> Duration: 6.843212402s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XXX.X.] duration: 3.454841125s trace: https://jaeger.dev.storj.io/trace/3d40a31b9816fab6 FAIL (2025-12-28 16:20:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454861874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.....................X.............X.........] duration: 32.778537956s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 32.778568243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.543123343s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:20:13 UTC) orbiter warning: context deadline exceeded Duration: 6.543152128s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X........X.X.X.X....XXX...X...X.......X....XXXXX.........X] duration: 30.004135747s trace: https://jaeger.dev.storj.io/trace/6221cb65a29c769b FAIL (2025-12-28 16:20:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 16:19:41.477959304 (552h50m41.349042339s) [upload=484702]: Starting upload Dec 28 16:19:41.478023667 (552h50m41.349106702s) [upload=484702]: Done starting upload Dec 28 16:19:41.478151841 (552h50m41.349234876s) [upload=484702]: Uploading segments... Dec 28 16:19:41.478456340 (552h50m41.349539385s) [upload=484702]: Got next segment. Inline: false Dec 28 16:19:41.478490636 (552h50m41.349573671s) [upload=484702]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:19:41.478496388 (552h50m41.349579433s) [upload=484702]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:19:41.725457381 (552h50m41.596540417s) [seg_pos={0 0}, upload=484702]: Begin upload segment... Dec 28 16:19:41.725496486 (552h50m41.596579531s) [seg_pos={0 0}, upload=484702]: Done begin upload segment. Dec 28 16:19:41.725524059 (552h50m41.596607094s) [noise=true , address=45.91.116.6:4976 , piece_id=ZTY24CLXXPKEZFYCSVLEEWEWCDYSCW7NCSV2NUP46Q2T7V2GFGBQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.725551511 (552h50m41.596634556s) [noise=true , address=145.79.227.16:4912 , piece_id=7FG7MRV7SP7CBDJXG6GSYMALXDB35RHGUVIC4Y6OCFS2MK6NV3LA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.726206979 (552h50m41.597290024s) [noise=true , address=145.79.227.15:4874 , piece_id=LUB5T7WNOEDSHUROEK2NTBFGY37P4JYAJHVN3PPMIHZ3D2LLX37Q, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.727712476 (552h50m41.598795511s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728009310 (552h50m41.599092356s) [noise=true , address=145.79.227.47:4265 , piece_id=5FWNDQJA7CWQTDO2R3RW3APWKKAOYVXJL4MIIWDKLPE2NHTOFOAQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728304032 (552h50m41.599387067s) [noise=true , address=45.91.116.7:4009 , piece_id=TUOAYAJ5T54RZWFKRKMK3M77TP4BG2BZ5ZAXE3FDIF2Q4TAN6L2Q, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728621737 (552h50m41.599704782s) [noise=true , address=45.91.116.13:4153 , piece_id=2YNDCGVX5NF2GOKEI4FL4YVI6HR2CEPPPJBRJV3VZEURTFXR2ZMA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.728939181 (552h50m41.600022216s) [noise=true , address=45.91.116.14:4189 , piece_id=BTSWH5ZRVXMWSK6GNZC4ZBIALJYCRBXAOPMNAQSYPTHPBQQVHMSQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729068398 (552h50m41.600151443s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729228292 (552h50m41.600311327s) [noise=true , address=145.79.227.38:4977 , piece_id=LW6NZHQINWAZ4BLHYTBWBUL7NCSYXBMPC7SBO5BRW5DKGTOZOIJQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729455825 (552h50m41.60053887s) [noise=true , address=145.79.227.46:4603 , piece_id=O552YHMQL2DYQLS2PEMZTHCCU3SYSCNOKVZOCRIDJ5K2Z75JPEVA, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729493387 (552h50m41.600576432s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.729870485 (552h50m41.60095352s) [noise=true , address=145.79.227.14:4838 , piece_id=NGHELVUBZ6KO2IMLG64BPVNLCMKROZAFFJ4OWE6C6PDNM4QRWRLQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.730441833 (552h50m41.601524878s) [noise=true , address=145.79.227.7:4294 , piece_id=N6KQ6PKWQ7OEORBA3TZAPTTDGEKPMLSFELAJHHLBPDF2OZMRY7NQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.730926756 (552h50m41.602009801s) [noise=true , address=145.79.227.9:4366 , piece_id=OIF7RU62XF5XHBP4M64GU2OBYRRAOLXAHCYXGT54O726OJDMFN2A, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:41.731333130 (552h50m41.602416165s) [noise=true , address=145.79.227.48:4303 , piece_id=FDJYNMU57R6XF4QUK32KFNIQX5TM3B6SSZKFDTFXR6J6BE6E6KIQ, seg_pos={0 0}, upload=484702]: Uploading piece... Dec 28 16:19:43.439258955 (552h50m43.310342s) [upload=484702]: Next returned nil segment Dec 28 16:19:43.439502248 (552h50m43.310585323s) [upload=484702]: Waiting for error group managing segments... Dec 28 16:19:43.726760847 (552h50m43.597843892s) [noise=true , address=145.79.227.9:4366 , piece_id=OIF7RU62XF5XHBP4M64GU2OBYRRAOLXAHCYXGT54O726OJDMFN2A, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.726797166 (552h50m43.597880211s) [noise=true , address=145.79.227.7:4294 , piece_id=N6KQ6PKWQ7OEORBA3TZAPTTDGEKPMLSFELAJHHLBPDF2OZMRY7NQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.734570077 (552h50m43.605653112s) [noise=true , address=145.79.227.46:4603 , piece_id=O552YHMQL2DYQLS2PEMZTHCCU3SYSCNOKVZOCRIDJ5K2Z75JPEVA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.738146557 (552h50m43.609229602s) [noise=true , address=145.79.227.48:4303 , piece_id=FDJYNMU57R6XF4QUK32KFNIQX5TM3B6SSZKFDTFXR6J6BE6E6KIQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.748516273 (552h50m43.619599358s) [noise=true , address=145.79.227.38:4977 , piece_id=LW6NZHQINWAZ4BLHYTBWBUL7NCSYXBMPC7SBO5BRW5DKGTOZOIJQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.752687726 (552h50m43.623770791s) [noise=true , address=145.79.227.47:4265 , piece_id=5FWNDQJA7CWQTDO2R3RW3APWKKAOYVXJL4MIIWDKLPE2NHTOFOAQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.756612580 (552h50m43.627695625s) [noise=true , address=145.79.227.15:4874 , piece_id=LUB5T7WNOEDSHUROEK2NTBFGY37P4JYAJHVN3PPMIHZ3D2LLX37Q, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.763676310 (552h50m43.634759355s) [noise=true , address=145.79.227.14:4838 , piece_id=NGHELVUBZ6KO2IMLG64BPVNLCMKROZAFFJ4OWE6C6PDNM4QRWRLQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:43.768023057 (552h50m43.639106092s) [noise=true , address=145.79.227.16:4912 , piece_id=7FG7MRV7SP7CBDJXG6GSYMALXDB35RHGUVIC4Y6OCFS2MK6NV3LA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.083729315 (552h50m43.95481235s) [noise=true , address=45.91.116.13:4153 , piece_id=2YNDCGVX5NF2GOKEI4FL4YVI6HR2CEPPPJBRJV3VZEURTFXR2ZMA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.084971881 (552h50m43.956054926s) [noise=true , address=45.91.116.7:4009 , piece_id=TUOAYAJ5T54RZWFKRKMK3M77TP4BG2BZ5ZAXE3FDIF2Q4TAN6L2Q, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.163872059 (552h50m44.034955104s) [noise=true , address=45.91.116.14:4189 , piece_id=BTSWH5ZRVXMWSK6GNZC4ZBIALJYCRBXAOPMNAQSYPTHPBQQVHMSQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.666546576 (552h50m44.537629611s) [noise=true , address=45.91.116.6:4976 , piece_id=ZTY24CLXXPKEZFYCSVLEEWEWCDYSCW7NCSV2NUP46Q2T7V2GFGBQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: <nil> Dec 28 16:19:44.666569931 (552h50m44.537652966s) [seg_pos={0 0}, upload=484702]: Segment reached optimal threshold of 13 pieces. Dec 28 16:19:44.666870052 (552h50m44.537953107s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 16:19:44.666879230 (552h50m44.537962285s) [noise=true , address=145.79.227.30:4762 , piece_id=3CCPN5LWFMX6ATQSZZOGG3EA5YWHN4UZIWMRZHDPNYGP4BD2JI4A, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667143703 (552h50m44.538226758s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 16:19:44.667162900 (552h50m44.538245935s) [noise=true , address=45.91.116.4:4940 , piece_id=HCZZSOEC2X2GWAX5Z7OSR5LB7XTLK6WYZN3YGMCRSZEYZCW2SUSA, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667235388 (552h50m44.538318433s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50888->45.91.116.8:4046: use of closed network connection Dec 28 16:19:44.667245407 (552h50m44.538328452s) [noise=true , address=45.91.116.8:4046 , piece_id=2I7OG4QDAZU7FNMGH2JX4K6EAHQOSQOOSTFEFC4N36M4SPZ2GXBQ, seg_pos={0 0}, upload=484702]: Long tail cancellation due to optimal threshold Dec 28 16:19:44.667267780 (552h50m44.538350815s) [seg_pos={0 0}, upload=484702]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:19:44.667361729 (552h50m44.538444764s) [upload=484702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:19:44.667367830 (552h50m44.538450865s) [upload=484702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:19:45.021706490 (552h50m44.892789525s) [upload=484702]: Done uploading segments... Dec 28 16:19:45.021725065 (552h50m44.89280811s) [upload=484702]: Upload finished. err: <nil> Duration: 30.004173179s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XXX.X] duration: 5.001343645s trace: https://jaeger.dev.storj.io/trace/3d40a31b9816fab6 FAIL (2025-12-28 16:20:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001366508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XXX.] duration: 5.000422509s trace: https://jaeger.dev.storj.io/trace/3ee026f4160787ff FAIL (2025-12-28 16:19:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00048576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.834475234s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:19:45 UTC) orbiter warning: context deadline exceeded Duration: 5.834512154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 11.598034626s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:19:44 UTC) orbiter warning: context deadline exceeded Duration: 11.598060393s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.....................X.............X........] duration: 33.172762657s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:19:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 33.172780862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X........X.X.X.X....XXX...X...X.......X....XXXXX.........] duration: 6.13184709s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:19:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 6.131873651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XXX] duration: 5.000989081s trace: https://jaeger.dev.storj.io/trace/3ee026f4160787ff FAIL (2025-12-28 16:18:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001008838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X........X.X.X.X....XXX...X...X.......X....XXXXX........] duration: 8.109464588s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 8.109515594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.....................X.............X.......] duration: 46.251685072s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:18:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 46.251726301s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 13.090427554s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:18:43 UTC) orbiter warning: context deadline exceeded Duration: 13.090454125s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 10.435589717s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:18:38 UTC) orbiter warning: context deadline exceeded Duration: 10.435614434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.X........X.X.X.X....XXX...X...X.......X....XXXXX.......] duration: 6.202399355s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:17:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 6.202424894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..XX] duration: 5.000893867s trace: https://jaeger.dev.storj.io/trace/72bfd29fc2111533 FAIL (2025-12-28 16:17:51 UTC) orbiter warning: context deadline exceeded Duration: 5.000918604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 10.43641995s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:17:47 UTC) orbiter warning: context deadline exceeded Duration: 10.436479593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.829445768s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:17:34 UTC) orbiter warning: context deadline exceeded Duration: 5.829473351s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....................X.............X......] duration: 26.651645455s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:17:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 26.651660304s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 12.611301105s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:17:21 UTC) orbiter warning: context deadline exceeded Duration: 12.611380375s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 10.707503323s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:17:12 UTC) orbiter warning: context deadline exceeded Duration: 10.707521398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....................X.............X.....] duration: 46.514648102s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:16:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 46.514683839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X.X........X.X.X.X....XXX...X...X.......X....XXXXX......] duration: 6.199068299s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:16:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 6.199093707s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..X] duration: 5.000738462s trace: https://jaeger.dev.storj.io/trace/4819fc42641e2b60 FAIL (2025-12-28 16:16:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000761997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 10.826711159s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:16:24 UTC) orbiter warning: context deadline exceeded Duration: 10.826740053s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX...X.X........X.X.X.X....XXX...X...X.......X....XXXXX.....] duration: 6.620886156s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:16:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 6.620916874s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X..] duration: 5.001658493s trace: https://jaeger.dev.storj.io/trace/6b03d5a6fd9c353b FAIL (2025-12-28 16:16:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001699632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 11.120258351s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:16:01 UTC) orbiter warning: context deadline exceeded Duration: 11.120288808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.....................X.............X....] duration: 43.731402518s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:15:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 43.731435992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 10.669987792s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:15:38 UTC) orbiter warning: context deadline exceeded Duration: 10.670018922s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 11.402858749s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:15:20 UTC) orbiter warning: context deadline exceeded Duration: 11.402882044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX...X.X........X.X.X.X....XXX...X...X.......X....XXXXX....] duration: 6.198932172s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:15:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 6.198966058s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.708252632s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:14:52 UTC) orbiter warning: context deadline exceeded Duration: 6.708300383s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XX.XX.XXX..XX...XXX.....XX..XX.XX.XXXXX...X.X.XXXX..XXXXX.X.] duration: 5.001703839s trace: https://jaeger.dev.storj.io/trace/6b03d5a6fd9c353b FAIL (2025-12-28 16:14:52 UTC) orbiter warning: context deadline exceeded Duration: 5.001780515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX...X.X........X.X.X.X....XXX...X...X.......X....XXXXX...] duration: 6.06435922s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:14:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 6.064392665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....................X.............X...] duration: 23.767823671s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:14:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 23.767839662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.624607473s trace: https://jaeger.dev.storj.io/trace/60f84e043b838a87 FAIL (2025-12-28 16:14:21 UTC) orbiter warning: context deadline exceeded Duration: 8.62467898s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 11.715383019s trace: https://jaeger.dev.storj.io/trace/d64711046bc169e FAIL (2025-12-28 16:14:14 UTC) orbiter warning: context deadline exceeded Duration: 11.715493198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.XX...X.X........X.X.X.X....XXX...X...X.......X....XXXXX..] duration: 5.667694047s trace: https://jaeger.dev.storj.io/trace/ec7752b51cc2b18 FAIL (2025-12-28 16:14:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA) context deadline exceeded ======LOGS====== 16:12:36.883066971 (552h43m36.754150007s) [upload=484594]: Starting upload Dec 28 16:12:36.883133458 (552h43m36.754216513s) [upload=484594]: Done starting upload Dec 28 16:12:36.883355380 (552h43m36.754438415s) [upload=484594]: Uploading segments... Dec 28 16:12:36.883411979 (552h43m36.754495024s) [upload=484594]: Got next segment. Inline: false Dec 28 16:12:36.883441996 (552h43m36.754525031s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 16:12:36.883447176 (552h43m36.754530221s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 16:12:37.128904327 (552h43m36.999987362s) [seg_pos={0 0}, upload=484594]: Begin upload segment... Dec 28 16:12:37.128947609 (552h43m37.000030654s) [seg_pos={0 0}, upload=484594]: Done begin upload segment. Dec 28 16:12:37.128973248 (552h43m37.000056293s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129033703 (552h43m37.000116738s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129423836 (552h43m37.000506882s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.129722004 (552h43m37.000805039s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130107618 (552h43m37.001190653s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130468506 (552h43m37.001551551s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.130801349 (552h43m37.001884394s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131072155 (552h43m37.00215519s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.131377787 (552h43m37.002460822s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.132824251 (552h43m37.003907296s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.133314023 (552h43m37.004397058s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.134992310 (552h43m37.006075345s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.137653737 (552h43m37.008736772s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.140257996 (552h43m37.011341031s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.142275258 (552h43m37.013358323s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:37.143656969 (552h43m37.014740004s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Uploading piece... Dec 28 16:12:38.848192970 (552h43m38.719276005s) [upload=484594]: Next returned nil segment Dec 28 16:12:38.848205113 (552h43m38.719288148s) [upload=484594]: Waiting for error group managing segments... Dec 28 16:12:39.129772584 (552h43m39.000855619s) [noise=true , address=145.79.227.4:4186 , piece_id=6R7I3VJCHX6CCA4KJLZJIQMOKMHRYSX3TUWEXASUWU245UGOXSOQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.133592909 (552h43m39.004675954s) [noise=true , address=145.79.227.19:4079 , piece_id=74JBENPIF62RVJZLEH2KHT3LU5V44DBDQPESYD4CLYMOERYEBLTA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.135940339 (552h43m39.007023374s) [noise=true , address=145.79.227.5:4223 , piece_id=VIESSXE2TVJPO47OC67W767JCPPFCYJLBLRAP2RHUBTR5YQH4YEA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.138687981 (552h43m39.009771026s) [noise=true , address=145.79.227.7:4295 , piece_id=2YH5BBD3HJKIOTA6EW6KVZH6G6YQL7DOSV43I5LTIXHEARULHWMA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.139159759 (552h43m39.010242794s) [noise=true , address=145.79.227.3:4149 , piece_id=CDMHQPV5KZNWMMRDJJOXPBVCOLS6MM56NA2SASXZR2SX3GMNRHMQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.155001324 (552h43m39.026084369s) [noise=true , address=145.79.227.48:4302 , piece_id=OUWDXECDUSVJMZSMZ4KD2CNUK3SAHBDHXG2P7BAGMGEAAKW4HPXQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.172209592 (552h43m39.043292627s) [noise=true , address=145.79.227.6:4257 , piece_id=2L55BQIDAERUCY3ARXY4CBDRSLKTRHA6QZUKLHUC4TIFATGBXCZQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.190456236 (552h43m39.061539281s) [noise=true , address=145.79.227.27:4690 , piece_id=4GEAQ4VZ5DXS6M3ASEXQ623Q7KJ5QM2HJLX5R25NC3LG5YJSWHYA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.350453839 (552h43m39.221536884s) [noise=true , address=45.91.116.10:4081 , piece_id=JPWYSFOQUZOWMCUBEEQW47TRHGPRGK7TS7XLZS4L76SMMDZL2WPA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.367728652 (552h43m39.238811697s) [noise=true , address=77.90.31.10:4519 , piece_id=7GWNJIKVSZVCBEOKPXVD7ADKGXLGWAMVC3MPGQ52XECCKC7GZ3TA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.416401323 (552h43m39.287484368s) [noise=true , address=45.91.116.11:4117 , piece_id=VAOJYAELMON6LBTC2EJSSMVAUKP7LQCW7UKSH6QVGLKAJYQLHQXA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.420706802 (552h43m39.291789837s) [noise=true , address=77.90.31.17:4770 , piece_id=GBBETFKVASOM3CXZZIVNUXIE3VQT2YPOSQOSV6RNZI2QEQF5G2FQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778556638 (552h43m39.649639683s) [noise=true , address=77.90.31.14:4664 , piece_id=FFEMXM7EAULND3DZJCBAWJWCMQF2ODFGDLYAPCVXYD2H2IT3RTWA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: <nil> Dec 28 16:12:39.778579992 (552h43m39.649663038s) [seg_pos={0 0}, upload=484594]: Segment reached optimal threshold of 13 pieces. Dec 28 16:12:39.778796585 (552h43m39.64987962s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->45.91.116.5:4298: use of closed network connection Dec 28 16:12:39.778805251 (552h43m39.649888286s) [noise=true , address=45.91.116.5:4298 , piece_id=LKJ7EI4KGV7JAOCDAZWZZC42IUQWUDL2M64ACJG6CV3OELMKISSQ, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779076868 (552h43m39.650159903s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779085294 (552h43m39.650168329s) [noise=true , address=45.91.116.6:4975 , piece_id=CA7BA224BS6IRQURM5UEP55IXCHBXUZEIN3ARQFFRITQDJAWVFFA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779221123 (552h43m39.650304168s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 16:12:39.779228978 (552h43m39.650312013s) [noise=true , address=45.91.116.8:4045 , piece_id=WUDAOCTEKWSQBWND74YPJW7HMQ2FNMX74S5A7C5UFRQA5VNNRNAA, seg_pos={0 0}, upload=484594]: Long tail cancellation due to optimal threshold Dec 28 16:12:39.779250719 (552h43m39.650333764s) [seg_pos={0 0}, upload=484594]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 16:12:39.779329750 (552h43m39.650412785s) [upload=484594]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:12:39.779336142 (552h43m39.650419187s) [upload=484594]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:12:40.070395227 (552h43m39.941478272s) [upload=484594]: Done uploading segments... Dec 28 16:12:40.070418641 (552h43m39.941501686s) [upload=484594]: Upload finished. err: <nil> Duration: 5.667726178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.....................X.............X..] duration: 48.695506625s trace: https://jaeger.dev.storj.io/trace/33c2e56ed36e9eb FAIL (2025-12-28 16:14:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== SF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.634726081 (552h42m14.505809126s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.635872714 (552h42m14.506955759s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.636869642 (552h42m14.507952687s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638009703 (552h42m14.509092748s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.638962276 (552h42m14.510045321s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.640273263 (552h42m14.511356308s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.641060022 (552h42m14.512143057s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.643084697 (552h42m14.514167732s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.644950028 (552h42m14.516033063s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.646827633 (552h42m14.517910678s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.648642709 (552h42m14.519725744s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.650068845 (552h42m14.52115188s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:14.651725019 (552h42m14.522808054s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Uploading piece... Dec 28 16:11:15.973083496 (552h42m15.844166532s) [upload=484576]: Next returned nil segment Dec 28 16:11:15.973100710 (552h42m15.844183755s) [upload=484576]: Waiting for error group managing segments... Dec 28 16:11:16.334421655 (552h42m16.2055047s) [noise=true , address=64.49.13.137:4503 , piece_id=JOBVIGVKBJEKHAXNNRSSJLW5KAW2NQLJSMKKHLLX5QQFSQB53HNA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.463821304 (552h42m16.334904349s) [noise=true , address=64.49.13.138:4491 , piece_id=DXL4FQ4GUBO5GN3S7MCCLAFCQX7PXLK7IT37N4N5EUZRQLCZI5CA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.533473693 (552h42m16.404556728s) [noise=true , address=145.79.227.27:4692 , piece_id=3BAQTC5IO46NYP5S5KIBELMQXP554I2IE4SJR6YYCLTFIWEVNIPA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.683879431 (552h42m16.554962466s) [noise=true , address=77.90.31.15:4729 , piece_id=TUR6GL7G4V7PJKACX4VMGZPRVDKOHVAZZEYVXJQAGHVMKLR5SXWA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.786514555 (552h42m16.6575976s) [noise=true , address=77.90.31.19:4848 , piece_id=AUPG5H74WZZJTCVPCMU4H7RGB5VNW4IFZWWSH5JLZJ7BGUSBURHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.788695247 (552h42m16.659778282s) [noise=true , address=77.90.31.13:4645 , piece_id=ZQLHWH7CIOT235IBQNXFXPTSPZLO45XAT3ESY2NQALUDZRBXM3RA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.791809237 (552h42m16.662892272s) [noise=true , address=77.90.31.16:4750 , piece_id=GVVTDK4KVYPRAO4MVVYMN5NNGF4IQ5GKAJMXK2PT2PYN4TMWODMA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.796739825 (552h42m16.66782286s) [noise=true , address=77.90.31.18:4813 , piece_id=QASZMZSR35467SUZJL6FEXTJYGAFXAN43IRS5FVKTZZIWC46LJEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.797025059 (552h42m16.668108104s) [noise=true , address=77.90.31.17:4797 , piece_id=PSASC3UY5RBJEFQHQN5MQKWQ2S5HGOPDQJ3Z2QEQ76AVCARAZ4PA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.851173597 (552h42m16.722256642s) [noise=true , address=45.91.116.14:4208 , piece_id=P7QEOHFXRFFNIA675UFVA5RBE5RBTSLMPOM6OU4R5FRBHN7LDU4A, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:16.919951942 (552h42m16.791034987s) [noise=true , address=45.91.116.7:4027 , piece_id=NDYFBPSTPFKU733UHZSF3CLHX7KROPB6WCLCXPFSEJIVZSFRU3BA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.484481791 (552h42m17.355564846s) [noise=true , address=145.79.227.40:4074 , piece_id=DCIFNEM2QOOFOB5ERJU6DSCY7R4OIZC7MRWY36R6BPVL7FZTDCHQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.514368851 (552h42m17.385451886s) [noise=true , address=145.79.227.33:4904 , piece_id=WOJ2IAHIDDC6YUFV5CFSV4WX5FXSLLKPXW6AEP4EMYNNF265UOIA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516174189 (552h42m17.387257234s) [noise=true , address=145.79.227.34:4940 , piece_id=YXYFDKUYON4WB4CIO6EUUZTPZNKV23UHI24X6JUI2XC744YYM6OQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: <nil> Dec 28 16:11:17.516187004 (552h42m17.387270049s) [seg_pos={0 0}, upload=484576]: Segment reached optimal threshold of 14 pieces. Dec 28 16:11:17.516393587 (552h42m17.387476622s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QbJar7xfiCubCHQ4hTTG4UYNVrxQig1mpvBKeXoEBUzbCHsJT): protocol: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45960->145.79.227.19:4088: use of closed network connection Dec 28 16:11:17.516399789 (552h42m17.387482834s) [noise=true , address=145.79.227.19:4088 , piece_id=BTSUHKQGLBDLIMGP52DW2US26EBL6CRUGKK6DJVFYVPNUTABYUVQ, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516613515 (552h42m17.387696551s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51592->64.49.13.169:4716: use of closed network connection Dec 28 16:11:17.516623815 (552h42m17.38770685s) [noise=true , address=64.49.13.169:4716 , piece_id=L4MELC4D63ARUD3E6M753XYJBOL247GCDTRYQKN4PVBBVI7RO52Q, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516757960 (552h42m17.387841005s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42184->45.91.116.6:4993: use of closed network connection Dec 28 16:11:17.516763360 (552h42m17.387846406s) [noise=true , address=45.91.116.6:4993 , piece_id=MDQMAUSZAAXSRQLL3IILAAII5LWIB2NPO7IYIW7XCR6DTAKXWWEA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516920590 (552h42m17.388003625s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84): protocol: expected piece hash; context canceled; EOF Dec 28 16:11:17.516927032 (552h42m17.388010078s) [noise=true , address=145.79.227.43:4160 , piece_id=APZLUISB3MECO43FAG5VH4ALFQCFBCGU44Y7C3EYQ2SV24PJAVUA, seg_pos={0 0}, upload=484576]: Long tail cancellation due to optimal threshold Dec 28 16:11:17.516943904 (552h42m17.388026939s) [seg_pos={0 0}, upload=484576]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 16:11:17.517017415 (552h42m17.38810045s) [upload=484576]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 16:11:17.517021593 (552h42m17.388104638s) [upload=484576]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 16:11:17.810953478 (552h42m17.682036523s) [upload=484576]: Done uploading segments... Dec 28 16:11:17.810971864 (552h42m17.682054899s) [upload=484576]: Upload finished. err: <nil> Duration: 48.695522205s core uplink us-select-1 us1-placement-20

1
...
11322
11323
11324
11325
11326
...
18023