Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX.] duration: 5.705774881s trace: https://jaeger.dev.storj.io/trace/21fde371d273b7b9 FAIL (2025-12-29 12:41:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ) context deadline exceeded ======LOGS====== Dec 29 12:39:58.079293040 (573h10m57.950376086s) [upload=502495]: Starting upload Dec 29 12:39:58.079401347 (573h10m57.950484382s) [upload=502495]: Done starting upload Dec 29 12:39:58.079483573 (573h10m57.950566618s) [upload=502495]: Uploading segments... Dec 29 12:39:58.079697892 (573h10m57.950780927s) [upload=502495]: Got next segment. Inline: false Dec 29 12:39:58.079745191 (573h10m57.950828236s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:58.079751894 (573h10m57.950834939s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:58.324037700 (573h10m58.195120745s) [seg_pos={0 0}, upload=502495]: Begin upload segment... Dec 29 12:39:58.324121059 (573h10m58.195204104s) [seg_pos={0 0}, upload=502495]: Done begin upload segment. Dec 29 12:39:58.324508537 (573h10m58.195591572s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.325309292 (573h10m58.196392337s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327037432 (573h10m58.198120467s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327819862 (573h10m58.198902897s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.328759832 (573h10m58.199842867s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.329778361 (573h10m58.200861396s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330143697 (573h10m58.201226732s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330486359 (573h10m58.201569394s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330888655 (573h10m58.20197169s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331282535 (573h10m58.20236557s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331646378 (573h10m58.202729414s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331878901 (573h10m58.202961936s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332011724 (573h10m58.203094759s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332324780 (573h10m58.203407815s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332627797 (573h10m58.203710832s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.338701111 (573h10m58.209784146s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:40:00.040558968 (573h10m59.911642013s) [upload=502495]: Next returned nil segment Dec 29 12:40:00.040622679 (573h10m59.911705724s) [upload=502495]: Waiting for error group managing segments... Dec 29 12:40:00.321872750 (573h11m0.192955785s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.325245653 (573h11m0.196328688s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.327096597 (573h11m0.198179643s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335103083 (573h11m0.206186128s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335909939 (573h11m0.206992984s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.336842656 (573h11m0.207925691s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.348942666 (573h11m0.220025711s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.356133288 (573h11m0.227216333s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.596712026 (573h11m0.467795071s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.621897211 (573h11m0.492980256s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.634502033 (573h11m0.505585078s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.697141127 (573h11m0.568224162s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908852586 (573h11m0.779935631s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908873036 (573h11m0.779956071s) [seg_pos={0 0}, upload=502495]: Segment reached optimal threshold of 13 pieces. Dec 29 12:40:00.909170142 (573h11m0.780253177s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909178027 (573h11m0.780261072s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909232610 (573h11m0.780315645s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection Dec 29 12:40:00.909248421 (573h11m0.780331466s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909337450 (573h11m0.780420485s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909352629 (573h11m0.780435704s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909382035 (573h11m0.78046508s) [seg_pos={0 0}, upload=502495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:40:00.909502023 (573h11m0.780585068s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:40:00.909508005 (573h11m0.78059104s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:40:01.205188494 (573h11m1.076286077s) [upload=502495]: Done uploading segments... Dec 29 12:40:01.205233319 (573h11m1.076316364s) [upload=502495]: Upload finished. err: <nil> Duration: 5.705791793s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 5.001147681s trace: https://jaeger.dev.storj.io/trace/5c453fcaae0bc3bf FAIL (2025-12-29 12:40:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001166898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.......X..........X..............] duration: 52.972212034s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 52.972231381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXXX] duration: 5.001685805s trace: https://jaeger.dev.storj.io/trace/624b21da5f99a626 FAIL (2025-12-29 12:40:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001710763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..XX] duration: 30.004040939s trace: https://jaeger.dev.storj.io/trace/21fde371d273b7b9 FAIL (2025-12-29 12:40:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: 6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ) context deadline exceeded ======LOGS====== Dec 29 12:39:58.079293040 (573h10m57.950376086s) [upload=502495]: Starting upload Dec 29 12:39:58.079401347 (573h10m57.950484382s) [upload=502495]: Done starting upload Dec 29 12:39:58.079483573 (573h10m57.950566618s) [upload=502495]: Uploading segments... Dec 29 12:39:58.079697892 (573h10m57.950780927s) [upload=502495]: Got next segment. Inline: false Dec 29 12:39:58.079745191 (573h10m57.950828236s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:58.079751894 (573h10m57.950834939s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:58.324037700 (573h10m58.195120745s) [seg_pos={0 0}, upload=502495]: Begin upload segment... Dec 29 12:39:58.324121059 (573h10m58.195204104s) [seg_pos={0 0}, upload=502495]: Done begin upload segment. Dec 29 12:39:58.324508537 (573h10m58.195591572s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.325309292 (573h10m58.196392337s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327037432 (573h10m58.198120467s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.327819862 (573h10m58.198902897s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.328759832 (573h10m58.199842867s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.329778361 (573h10m58.200861396s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330143697 (573h10m58.201226732s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330486359 (573h10m58.201569394s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.330888655 (573h10m58.20197169s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331282535 (573h10m58.20236557s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331646378 (573h10m58.202729414s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.331878901 (573h10m58.202961936s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332011724 (573h10m58.203094759s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332324780 (573h10m58.203407815s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.332627797 (573h10m58.203710832s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:39:58.338701111 (573h10m58.209784146s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Uploading piece... Dec 29 12:40:00.040558968 (573h10m59.911642013s) [upload=502495]: Next returned nil segment Dec 29 12:40:00.040622679 (573h10m59.911705724s) [upload=502495]: Waiting for error group managing segments... Dec 29 12:40:00.321872750 (573h11m0.192955785s) [noise=true , address=145.79.227.6:4259 , piece_id=IN3S5E3FZ7DSAF4PP6JSBNMSFZPIQL2LYH4VK2IYRV6MO6BXTFBA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.325245653 (573h11m0.196328688s) [noise=true , address=145.79.227.11:4732 , piece_id=XH3VKJFMRUICQAOL55CCVRIKSOZNFKOELTXXPGVBWV73UBFJKPYQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.327096597 (573h11m0.198179643s) [noise=true , address=145.79.227.42:4122 , piece_id=YUIQXL7EORDFJ5ALQ5AK6EIB7W2BVK25K5CYVTEBRA23YV43VWTQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335103083 (573h11m0.206186128s) [noise=true , address=145.79.227.38:4977 , piece_id=IIXS6JWAIRDLRHH24T7POQILIEB2DR3F3WEOWGLJK3ZDVALPOYHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.335909939 (573h11m0.206992984s) [noise=true , address=145.79.227.16:4911 , piece_id=HIKXTI3QWEB36ZRDN3JKY3EXMTPIEURY3NI3HBDEBTFNPLXEPTLA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.336842656 (573h11m0.207925691s) [noise=true , address=145.79.227.52:4446 , piece_id=SMWRXV7DXGUUDRGZ4UZS2NT5CPRTK4R2JDZZ7ID4B5OI65YWWJOA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.348942666 (573h11m0.220025711s) [noise=true , address=145.79.227.3:4149 , piece_id=ZPPSSXJH5QPBC5UREMQQRML2TLW6L6RSK7WJ5TDIBG5N5VBHGTBQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.356133288 (573h11m0.227216333s) [noise=true , address=145.79.227.13:4802 , piece_id=TDV4WFGSDNSGHS4OLWKFVK4GK3YD62RA3Q6CXH27DFRQ2WHARNGA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.596712026 (573h11m0.467795071s) [noise=true , address=45.91.116.5:4298 , piece_id=DV5IJ67OMZW6C4GHHYVF7KRJN76MTT4LCWCNPAXBTGB7VVX2ZTHA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.621897211 (573h11m0.492980256s) [noise=true , address=45.91.116.12:4226 , piece_id=BTT33OZFMS22BD2AVBNWYXE34OVCREDUQPAB5YWYTCMFINBUGWYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.634502033 (573h11m0.505585078s) [noise=true , address=77.90.31.19:4844 , piece_id=XIP3WOQLA4XRM472AJHOAVMICJLMAUD7OVNOLP3RG7HARFOU2DVQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.697141127 (573h11m0.568224162s) [noise=true , address=45.91.116.10:4082 , piece_id=CU5DA6PX4YVIFWK4J3425WNBCV62DHWO47JY6V5K6DALA54OGRKA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908852586 (573h11m0.779935631s) [noise=true , address=45.91.116.7:4012 , piece_id=6W5E24DBPHFKZUQQ6VH7IXF65ZGWVKQFXP5ITBSADFQA4DWW4WFA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: <nil> Dec 29 12:40:00.908873036 (573h11m0.779956071s) [seg_pos={0 0}, upload=502495]: Segment reached optimal threshold of 13 pieces. Dec 29 12:40:00.909170142 (573h11m0.780253177s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909178027 (573h11m0.780261072s) [noise=true , address=77.90.31.15:4697 , piece_id=CR3FVBY3IABGTT7VTJ7RZ3VAERQFFXBG2C6RNDQRZHECK6QTKOXA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909232610 (573h11m0.780315645s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41878->77.90.31.12:4589: use of closed network connection Dec 29 12:40:00.909248421 (573h11m0.780331466s) [noise=true , address=77.90.31.12:4589 , piece_id=6QWZ4J4JKVG6AZF67TJCTDFZ3BTSYNGYDR5JWUWY6CJTPDLSSMYA, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909337450 (573h11m0.780420485s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:40:00.909352629 (573h11m0.780435704s) [noise=true , address=45.91.116.6:4974 , piece_id=WIPNRFHYXBU4LUV2M53RC3F3UEPSSGHSV52U44PH5CBNU36O2CGQ, seg_pos={0 0}, upload=502495]: Long tail cancellation due to optimal threshold Dec 29 12:40:00.909382035 (573h11m0.78046508s) [seg_pos={0 0}, upload=502495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:40:00.909502023 (573h11m0.780585068s) [upload=502495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:40:00.909508005 (573h11m0.78059104s) [upload=502495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:40:01.205188494 (573h11m1.076286077s) [upload=502495]: Done uploading segments... Dec 29 12:40:01.205233319 (573h11m1.076316364s) [upload=502495]: Upload finished. err: <nil> Duration: 30.004067879s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.001459464s trace: https://jaeger.dev.storj.io/trace/7092646b240658b4 FAIL (2025-12-29 12:40:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001487318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.......X..........X.............] duration: 48.080279293s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:39:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 48.080307627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..X] duration: 30.00293109s trace: https://jaeger.dev.storj.io/trace/21c017eafb222c2f FAIL (2025-12-29 12:39:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: HKXTBGBIYUHEIVRYZAWZE74TAZ3KJNN5LFZWHRKMG3BXECXR6JCA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: UNNLHTGL5GCEKYNGRGB7LDFQDEXNECNDNVKHAZ3VSLVDENFJFRXA) context deadline exceeded ======LOGS====== Dec 29 12:39:13.395539621 (573h10m13.266622656s) [upload=502486]: Starting upload Dec 29 12:39:13.395617038 (573h10m13.266700073s) [upload=502486]: Done starting upload Dec 29 12:39:13.395762635 (573h10m13.26684567s) [upload=502486]: Uploading segments... Dec 29 12:39:13.396294368 (573h10m13.267377433s) [upload=502486]: Got next segment. Inline: false Dec 29 12:39:13.396364732 (573h10m13.267447767s) [upload=502486]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:39:13.396378027 (573h10m13.267461062s) [upload=502486]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:39:13.643573624 (573h10m13.514656659s) [seg_pos={0 0}, upload=502486]: Begin upload segment... Dec 29 12:39:13.643624971 (573h10m13.514708006s) [seg_pos={0 0}, upload=502486]: Done begin upload segment. Dec 29 12:39:13.643653726 (573h10m13.514736771s) [noise=true , address=77.90.31.16:4735 , piece_id=GVVYVKXSABYKTB2LJ4P3DR6WUGT43TWN64KUX5XAFA6XVFLLO77Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.643730041 (573h10m13.514813086s) [noise=true , address=145.79.227.47:4265 , piece_id=RXKFTRDQRUGFFLBH5F3CK3UDRDAAVSLRFHSXGRRIEPFUPUCFHJ4Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.644341977 (573h10m13.515425012s) [noise=true , address=145.79.227.42:4122 , piece_id=APA6XQNVMGXNHTILOJNAQIMVP6C55ZK6DYDLF6O355SQAMUJMVKQ, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.644565973 (573h10m13.515649018s) [noise=true , address=145.79.227.36:4519 , piece_id=6WJHDDBMWJA5JZPRBKQ7HSWYW5AYNS4DCOBDRDLKST22CXRRWIDA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.644836388 (573h10m13.515919423s) [noise=true , address=77.90.31.11:4555 , piece_id=HKXTBGBIYUHEIVRYZAWZE74TAZ3KJNN5LFZWHRKMG3BXECXR6JCA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.645054754 (573h10m13.516137789s) [noise=true , address=145.79.227.3:4150 , piece_id=CY5SJKOKFGMA36SFMLLVMRSSAE2IYNB6R6XDQGXJALKUU2AHQK4Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.645249375 (573h10m13.51633241s) [noise=true , address=45.91.116.12:4226 , piece_id=UNNLHTGL5GCEKYNGRGB7LDFQDEXNECNDNVKHAZ3VSLVDENFJFRXA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.645743045 (573h10m13.51682608s) [noise=true , address=145.79.227.51:4409 , piece_id=CKNQ5QHUM6UKBUDKYO3GXTI2YSVKQ6HD5G3VVA2YWFJH5GYDAKRA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.646205405 (573h10m13.51728844s) [noise=true , address=145.79.227.6:4258 , piece_id=A3H7P5ZSFWO5R2HQZPB3BQK7MN5QJQHBN6L2P5LRWV42T6OZLCJA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.646634112 (573h10m13.517717147s) [noise=true , address=45.91.116.4:4940 , piece_id=FWN5EZMVXNH7636CCKRWSSVSCZQIYUK6YLS7JJXLY4QKZOTCAR3Q, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647025296 (573h10m13.518108341s) [noise=true , address=77.90.31.21:4877 , piece_id=W56E4BPS7NVKJ2UH2HEHVFEI3TVAZZT27FKWWEBCXCHDNGQPDHLQ, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647364513 (573h10m13.518447548s) [noise=true , address=45.91.116.13:4155 , piece_id=CGPCS7FFKTLNQBAAWEKVFUTVPEP47LVPYYSWOWUQMOWWVR2KBDRA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647730359 (573h10m13.518813394s) [noise=true , address=145.79.227.26:4654 , piece_id=UHPHLT5TGTH7M2IOC6HNSIXN4XBNYQ5TYRXX3P4BM454B367AMGQ, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.647882588 (573h10m13.518965623s) [noise=true , address=145.79.227.21:4473 , piece_id=OGWM4YESCYKHYQPSLNG45G53EXAALTPOX3INIRJJPMG5PEYCBVFA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.648511095 (573h10m13.51959413s) [noise=true , address=145.79.227.32:4834 , piece_id=4E3WBRVNVZ5VM4SINUMTSFWX34WKJOJBY7CF5OCXTS6LE4EHSRWA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:13.648872373 (573h10m13.519955418s) [noise=true , address=145.79.227.4:4187 , piece_id=X2LMTFKJETIIEJVFILFPM6ANDDE4NBEL7QBRB4KNAQG7TT4FFVNA, seg_pos={0 0}, upload=502486]: Uploading piece... Dec 29 12:39:15.338610873 (573h10m15.209693908s) [upload=502486]: Next returned nil segment Dec 29 12:39:15.338624509 (573h10m15.209707544s) [upload=502486]: Waiting for error group managing segments... Dec 29 12:39:15.630861170 (573h10m15.501944205s) [noise=true , address=145.79.227.4:4187 , piece_id=X2LMTFKJETIIEJVFILFPM6ANDDE4NBEL7QBRB4KNAQG7TT4FFVNA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.632015588 (573h10m15.503098623s) [noise=true , address=145.79.227.21:4473 , piece_id=OGWM4YESCYKHYQPSLNG45G53EXAALTPOX3INIRJJPMG5PEYCBVFA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.633438318 (573h10m15.504521353s) [noise=true , address=145.79.227.36:4519 , piece_id=6WJHDDBMWJA5JZPRBKQ7HSWYW5AYNS4DCOBDRDLKST22CXRRWIDA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.633740383 (573h10m15.504823418s) [noise=true , address=145.79.227.51:4409 , piece_id=CKNQ5QHUM6UKBUDKYO3GXTI2YSVKQ6HD5G3VVA2YWFJH5GYDAKRA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.637589111 (573h10m15.508672146s) [noise=true , address=145.79.227.26:4654 , piece_id=UHPHLT5TGTH7M2IOC6HNSIXN4XBNYQ5TYRXX3P4BM454B367AMGQ, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.645457494 (573h10m15.516540539s) [noise=true , address=145.79.227.6:4258 , piece_id=A3H7P5ZSFWO5R2HQZPB3BQK7MN5QJQHBN6L2P5LRWV42T6OZLCJA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.650268955 (573h10m15.521351991s) [noise=true , address=145.79.227.32:4834 , piece_id=4E3WBRVNVZ5VM4SINUMTSFWX34WKJOJBY7CF5OCXTS6LE4EHSRWA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.654572921 (573h10m15.525655956s) [noise=true , address=145.79.227.3:4150 , piece_id=CY5SJKOKFGMA36SFMLLVMRSSAE2IYNB6R6XDQGXJALKUU2AHQK4Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.659751952 (573h10m15.530834987s) [noise=true , address=145.79.227.42:4122 , piece_id=APA6XQNVMGXNHTILOJNAQIMVP6C55ZK6DYDLF6O355SQAMUJMVKQ, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.667170017 (573h10m15.538253063s) [noise=true , address=145.79.227.47:4265 , piece_id=RXKFTRDQRUGFFLBH5F3CK3UDRDAAVSLRFHSXGRRIEPFUPUCFHJ4Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.905858154 (573h10m15.776941199s) [noise=true , address=77.90.31.16:4735 , piece_id=GVVYVKXSABYKTB2LJ4P3DR6WUGT43TWN64KUX5XAFA6XVFLLO77Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.935432518 (573h10m15.806515553s) [noise=true , address=45.91.116.12:4226 , piece_id=UNNLHTGL5GCEKYNGRGB7LDFQDEXNECNDNVKHAZ3VSLVDENFJFRXA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.969983028 (573h10m15.841066063s) [noise=true , address=77.90.31.11:4555 , piece_id=HKXTBGBIYUHEIVRYZAWZE74TAZ3KJNN5LFZWHRKMG3BXECXR6JCA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: <nil> Dec 29 12:39:15.970001262 (573h10m15.841084307s) [seg_pos={0 0}, upload=502486]: Segment reached optimal threshold of 13 pieces. Dec 29 12:39:15.970202896 (573h10m15.841285961s) [noise=true , address=45.91.116.13:4155 , piece_id=CGPCS7FFKTLNQBAAWEKVFUTVPEP47LVPYYSWOWUQMOWWVR2KBDRA, seg_pos={0 0}, upload=502486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 12:39:15.970214057 (573h10m15.841297092s) [noise=true , address=45.91.116.13:4155 , piece_id=CGPCS7FFKTLNQBAAWEKVFUTVPEP47LVPYYSWOWUQMOWWVR2KBDRA, seg_pos={0 0}, upload=502486]: Long tail cancellation due to optimal threshold Dec 29 12:39:15.970440639 (573h10m15.841523694s) [noise=true , address=77.90.31.21:4877 , piece_id=W56E4BPS7NVKJ2UH2HEHVFEI3TVAZZT27FKWWEBCXCHDNGQPDHLQ, seg_pos={0 0}, upload=502486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 29 12:39:15.970450317 (573h10m15.841533352s) [noise=true , address=77.90.31.21:4877 , piece_id=W56E4BPS7NVKJ2UH2HEHVFEI3TVAZZT27FKWWEBCXCHDNGQPDHLQ, seg_pos={0 0}, upload=502486]: Long tail cancellation due to optimal threshold Dec 29 12:39:15.970454796 (573h10m15.841537831s) [noise=true , address=45.91.116.4:4940 , piece_id=FWN5EZMVXNH7636CCKRWSSVSCZQIYUK6YLS7JJXLY4QKZOTCAR3Q, seg_pos={0 0}, upload=502486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 29 12:39:15.970465727 (573h10m15.841548782s) [noise=true , address=45.91.116.4:4940 , piece_id=FWN5EZMVXNH7636CCKRWSSVSCZQIYUK6YLS7JJXLY4QKZOTCAR3Q, seg_pos={0 0}, upload=502486]: Long tail cancellation due to optimal threshold Dec 29 12:39:15.970509801 (573h10m15.841592846s) [seg_pos={0 0}, upload=502486]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:39:15.970640600 (573h10m15.841723635s) [upload=502486]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:39:15.970648505 (573h10m15.84173154s) [upload=502486]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:39:16.267526194 (573h10m16.138609229s) [upload=502486]: Done uploading segments... Dec 29 12:39:16.267543817 (573h10m16.138626852s) [upload=502486]: Upload finished. err: <nil> Duration: 30.002956658s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [X..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXXX] duration: 5.001239775s trace: https://jaeger.dev.storj.io/trace/3732de33f4a19986 FAIL (2025-12-29 12:39:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001268901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX..] duration: 6.080351511s trace: https://jaeger.dev.storj.io/trace/500a2d0c59c5b5ec FAIL (2025-12-29 12:38:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ) context deadline exceeded ======LOGS====== Dec 29 12:36:54.758223142 (573h7m54.629306188s) [upload=502450]: Starting upload Dec 29 12:36:54.758313345 (573h7m54.62939639s) [upload=502450]: Done starting upload Dec 29 12:36:54.758408967 (573h7m54.629492002s) [upload=502450]: Uploading segments... Dec 29 12:36:54.758586685 (573h7m54.62966973s) [upload=502450]: Got next segment. Inline: false Dec 29 12:36:54.758621131 (573h7m54.629704176s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:36:54.758626170 (573h7m54.629709215s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:36:55.009502065 (573h7m54.8805851s) [seg_pos={0 0}, upload=502450]: Begin upload segment... Dec 29 12:36:55.009547962 (573h7m54.880630997s) [seg_pos={0 0}, upload=502450]: Done begin upload segment. Dec 29 12:36:55.009574803 (573h7m54.880657848s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.009702536 (573h7m54.880785592s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010481970 (573h7m54.881565015s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010949090 (573h7m54.882032125s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011378528 (573h7m54.882461573s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011806102 (573h7m54.882889137s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.012261579 (573h7m54.883344624s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013384478 (573h7m54.884467523s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013711470 (573h7m54.884794515s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.014701555 (573h7m54.88578459s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.015806499 (573h7m54.886889544s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.017025931 (573h7m54.888108966s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018296120 (573h7m54.889379165s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018681073 (573h7m54.889764118s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.021355326 (573h7m54.892438371s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.022526836 (573h7m54.893609881s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:56.711504397 (573h7m56.582587442s) [upload=502450]: Next returned nil segment Dec 29 12:36:56.711543431 (573h7m56.582626466s) [upload=502450]: Waiting for error group managing segments... Dec 29 12:36:56.993515769 (573h7m56.864598804s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:56.998999490 (573h7m56.870082525s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002018459 (573h7m56.873101524s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002541535 (573h7m56.87362459s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.004427697 (573h7m56.875510762s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006494572 (573h7m56.877577617s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006585926 (573h7m56.877668971s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023263683 (573h7m56.894346718s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023978755 (573h7m56.8950618s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.026445592 (573h7m56.897528627s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.039208685 (573h7m56.91029172s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.200168957 (573h7m57.071251992s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264465446 (573h7m57.135548481s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264483962 (573h7m57.135566997s) [seg_pos={0 0}, upload=502450]: Segment reached optimal threshold of 13 pieces. Dec 29 12:36:57.264656921 (573h7m57.135739957s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264663303 (573h7m57.135746339s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264799012 (573h7m57.135882067s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264807869 (573h7m57.135890914s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264981509 (573h7m57.136064554s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:45708->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:36:57.264992690 (573h7m57.136075745s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.265053376 (573h7m57.136136421s) [seg_pos={0 0}, upload=502450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:36:57.265158436 (573h7m57.136241471s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:36:57.265163325 (573h7m57.13624636s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:57.562292653 (573h7m57.433375698s) [upload=502450]: Done uploading segments... Dec 29 12:36:57.562315276 (573h7m57.433398321s) [upload=502450]: Upload finished. err: <nil> Duration: 6.080380446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.......X..........X............] duration: 35.346547626s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 35.346561091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXXX] duration: 5.000608065s trace: https://jaeger.dev.storj.io/trace/6d7506f73ad42c99 FAIL (2025-12-29 12:38:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000634868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XXX] duration: 5.001160123s trace: https://jaeger.dev.storj.io/trace/2ccf3ee2f427d57 FAIL (2025-12-29 12:38:01 UTC) orbiter warning: context deadline exceeded Duration: 5.001187536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.......X..........X...........] duration: 43.877802523s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:37:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 43.877817852s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX.] duration: 5.756860128s trace: https://jaeger.dev.storj.io/trace/500a2d0c59c5b5ec FAIL (2025-12-29 12:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ) context deadline exceeded ======LOGS====== Dec 29 12:36:54.758223142 (573h7m54.629306188s) [upload=502450]: Starting upload Dec 29 12:36:54.758313345 (573h7m54.62939639s) [upload=502450]: Done starting upload Dec 29 12:36:54.758408967 (573h7m54.629492002s) [upload=502450]: Uploading segments... Dec 29 12:36:54.758586685 (573h7m54.62966973s) [upload=502450]: Got next segment. Inline: false Dec 29 12:36:54.758621131 (573h7m54.629704176s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:36:54.758626170 (573h7m54.629709215s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:36:55.009502065 (573h7m54.8805851s) [seg_pos={0 0}, upload=502450]: Begin upload segment... Dec 29 12:36:55.009547962 (573h7m54.880630997s) [seg_pos={0 0}, upload=502450]: Done begin upload segment. Dec 29 12:36:55.009574803 (573h7m54.880657848s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.009702536 (573h7m54.880785592s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010481970 (573h7m54.881565015s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010949090 (573h7m54.882032125s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011378528 (573h7m54.882461573s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011806102 (573h7m54.882889137s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.012261579 (573h7m54.883344624s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013384478 (573h7m54.884467523s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013711470 (573h7m54.884794515s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.014701555 (573h7m54.88578459s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.015806499 (573h7m54.886889544s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.017025931 (573h7m54.888108966s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018296120 (573h7m54.889379165s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018681073 (573h7m54.889764118s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.021355326 (573h7m54.892438371s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.022526836 (573h7m54.893609881s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:56.711504397 (573h7m56.582587442s) [upload=502450]: Next returned nil segment Dec 29 12:36:56.711543431 (573h7m56.582626466s) [upload=502450]: Waiting for error group managing segments... Dec 29 12:36:56.993515769 (573h7m56.864598804s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:56.998999490 (573h7m56.870082525s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002018459 (573h7m56.873101524s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002541535 (573h7m56.87362459s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.004427697 (573h7m56.875510762s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006494572 (573h7m56.877577617s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006585926 (573h7m56.877668971s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023263683 (573h7m56.894346718s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023978755 (573h7m56.8950618s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.026445592 (573h7m56.897528627s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.039208685 (573h7m56.91029172s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.200168957 (573h7m57.071251992s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264465446 (573h7m57.135548481s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264483962 (573h7m57.135566997s) [seg_pos={0 0}, upload=502450]: Segment reached optimal threshold of 13 pieces. Dec 29 12:36:57.264656921 (573h7m57.135739957s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264663303 (573h7m57.135746339s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264799012 (573h7m57.135882067s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264807869 (573h7m57.135890914s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264981509 (573h7m57.136064554s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:45708->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:36:57.264992690 (573h7m57.136075745s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.265053376 (573h7m57.136136421s) [seg_pos={0 0}, upload=502450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:36:57.265158436 (573h7m57.136241471s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:36:57.265163325 (573h7m57.13624636s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:57.562292653 (573h7m57.433375698s) [upload=502450]: Done uploading segments... Dec 29 12:36:57.562315276 (573h7m57.433398321s) [upload=502450]: Upload finished. err: <nil> Duration: 5.756885976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXXX] duration: 30.002738193s trace: https://jaeger.dev.storj.io/trace/500a2d0c59c5b5ec FAIL (2025-12-29 12:37:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ) context deadline exceeded ======LOGS====== Dec 29 12:36:54.758223142 (573h7m54.629306188s) [upload=502450]: Starting upload Dec 29 12:36:54.758313345 (573h7m54.62939639s) [upload=502450]: Done starting upload Dec 29 12:36:54.758408967 (573h7m54.629492002s) [upload=502450]: Uploading segments... Dec 29 12:36:54.758586685 (573h7m54.62966973s) [upload=502450]: Got next segment. Inline: false Dec 29 12:36:54.758621131 (573h7m54.629704176s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:36:54.758626170 (573h7m54.629709215s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:36:55.009502065 (573h7m54.8805851s) [seg_pos={0 0}, upload=502450]: Begin upload segment... Dec 29 12:36:55.009547962 (573h7m54.880630997s) [seg_pos={0 0}, upload=502450]: Done begin upload segment. Dec 29 12:36:55.009574803 (573h7m54.880657848s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.009702536 (573h7m54.880785592s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010481970 (573h7m54.881565015s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.010949090 (573h7m54.882032125s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011378528 (573h7m54.882461573s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.011806102 (573h7m54.882889137s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.012261579 (573h7m54.883344624s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013384478 (573h7m54.884467523s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.013711470 (573h7m54.884794515s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.014701555 (573h7m54.88578459s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.015806499 (573h7m54.886889544s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.017025931 (573h7m54.888108966s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018296120 (573h7m54.889379165s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.018681073 (573h7m54.889764118s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.021355326 (573h7m54.892438371s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:55.022526836 (573h7m54.893609881s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Uploading piece... Dec 29 12:36:56.711504397 (573h7m56.582587442s) [upload=502450]: Next returned nil segment Dec 29 12:36:56.711543431 (573h7m56.582626466s) [upload=502450]: Waiting for error group managing segments... Dec 29 12:36:56.993515769 (573h7m56.864598804s) [noise=true , address=145.79.227.37:4566 , piece_id=WMAXPAYJQ6D57FTNPJ7UQDOCNV4RQAZIZ7XLL4XTOQBFHFX7MP6Q, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:56.998999490 (573h7m56.870082525s) [noise=true , address=145.79.227.14:4840 , piece_id=GFLHTJYOYZLSYO7Y4NASF53P6WZJKERKUDIG3IN7NHGL7Y5TIBSQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002018459 (573h7m56.873101524s) [noise=true , address=145.79.227.6:4257 , piece_id=IY7AK5N5AH26KTLPPWPNCVYIHO2KGQPX5V334EI6AQ4ZLXLVTSBQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.002541535 (573h7m56.87362459s) [noise=true , address=145.79.227.24:4582 , piece_id=A53AHESJPWXLPZGBS7IQNUQPEAXUPHCGLPUD3H7XHFWH4Y6GB6TQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.004427697 (573h7m56.875510762s) [noise=true , address=145.79.227.3:4151 , piece_id=A2EENCSAR6PH22OL7MI7LF35MWUVHYITJX7Y4UZP4EMN6HFZXVJA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006494572 (573h7m56.877577617s) [noise=true , address=145.79.227.32:4836 , piece_id=CVHL5TIGJO6BITRNSJ5ZGZDQZPO3D7X7L7TCWFKOKK4BDURAXUCQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.006585926 (573h7m56.877668971s) [noise=true , address=145.79.227.17:4971 , piece_id=O5FSPZHF4KFZAYLS4GB24PG2JXYRWRA6T4KAXM5ZR7X4MIJI7IUQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023263683 (573h7m56.894346718s) [noise=true , address=145.79.227.18:4006 , piece_id=WP3HU6PJ7W7DAYRHQREEL42JTN6Q6ZR5XHQUDPC2N3HIS64CABTA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.023978755 (573h7m56.8950618s) [noise=true , address=145.79.227.50:4374 , piece_id=ZWI2LXLPGKFOFDROSXIUK2RH42SIGKKXB4I7EE2I52MXZALFVBDQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.026445592 (573h7m56.897528627s) [noise=true , address=145.79.227.38:4977 , piece_id=INA7GEZGKAEDYLE6LDUKFJR6QM7STG5AVVKBXQFGOTGXD5AQOEUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.039208685 (573h7m56.91029172s) [noise=true , address=145.79.227.36:4519 , piece_id=KNHDIQFXPS6QE6NHRFUEP6P53NHG6U3DND34FAO3MH2WRI45ODGQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.200168957 (573h7m57.071251992s) [noise=true , address=77.90.31.12:4589 , piece_id=MSHLBIQYVLQ4FWFLBKPZTXU7O7VQD6YIPBED5WNSLW5ORD3UZ55A, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264465446 (573h7m57.135548481s) [noise=true , address=45.91.116.13:4155 , piece_id=75M2ZOIGQPDO7EQ24OBOOB2FGTDPTT74BQIERHPSCQU2SNFFWXNQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: <nil> Dec 29 12:36:57.264483962 (573h7m57.135566997s) [seg_pos={0 0}, upload=502450]: Segment reached optimal threshold of 13 pieces. Dec 29 12:36:57.264656921 (573h7m57.135739957s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264663303 (573h7m57.135746339s) [noise=true , address=145.79.227.49:4337 , piece_id=ZZ4VXS5PJ2JUTLB35ICCJYQY6JIIUM3GREXPHLW55VUZL2BIKMIQ, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264799012 (573h7m57.135882067s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 12:36:57.264807869 (573h7m57.135890914s) [noise=true , address=145.79.227.30:4761 , piece_id=LHCAVI2MWXDWEIESJ3U7JOVJJIGYZM7RNS7TMQ2IUM4V67PKX6VA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.264981509 (573h7m57.136064554s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:45708->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:36:57.264992690 (573h7m57.136075745s) [noise=true , address=45.91.116.9:4264 , piece_id=IVTKOWFT6PIYIHFBAHQRGF6MAAPHTHVIOMK6BBZTCHHV5E72GMUA, seg_pos={0 0}, upload=502450]: Long tail cancellation due to optimal threshold Dec 29 12:36:57.265053376 (573h7m57.136136421s) [seg_pos={0 0}, upload=502450]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:36:57.265158436 (573h7m57.136241471s) [upload=502450]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:36:57.265163325 (573h7m57.13624636s) [upload=502450]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:57.562292653 (573h7m57.433375698s) [upload=502450]: Done uploading segments... Dec 29 12:36:57.562315276 (573h7m57.433398321s) [upload=502450]: Upload finished. err: <nil> Duration: 30.002760225s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.XX] duration: 5.001493278s trace: https://jaeger.dev.storj.io/trace/4c872b9f771e22c8 FAIL (2025-12-29 12:37:19 UTC) orbiter warning: context deadline exceeded Duration: 5.001518305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.......X..........X..........] duration: 50.342683992s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 50.342711144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [..XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.X] duration: 5.001100594s trace: https://jaeger.dev.storj.io/trace/1bd9f56a9e88d2ee FAIL (2025-12-29 12:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001113318s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XXX] duration: 30.003065887s trace: https://jaeger.dev.storj.io/trace/3b55af95361e57c1 FAIL (2025-12-29 12:36:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =502439]: Starting upload Dec 29 12:35:56.314641001 (573h6m56.185724036s) [upload=502439]: Done starting upload Dec 29 12:35:56.314728368 (573h6m56.185811413s) [upload=502439]: Uploading segments... Dec 29 12:35:56.314888352 (573h6m56.185971397s) [upload=502439]: Got next segment. Inline: false Dec 29 12:35:56.314932406 (573h6m56.186015441s) [upload=502439]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:35:56.314938959 (573h6m56.186021994s) [upload=502439]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:35:56.567188299 (573h6m56.438271334s) [seg_pos={0 0}, upload=502439]: Begin upload segment... Dec 29 12:35:56.567242431 (573h6m56.438325466s) [seg_pos={0 0}, upload=502439]: Done begin upload segment. Dec 29 12:35:56.567276065 (573h6m56.43835911s) [noise=true , address=145.79.227.36:4518 , piece_id=755BMSS3R3L65EKO3VGQRYJHLEFYCGXT27YVJCAHUJRIEPEPPJVA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567302065 (573h6m56.4383851s) [noise=true , address=145.79.227.3:4150 , piece_id=C2SKGSU7DBMKEGNW6SQ7ESNITDHVTNGHE4JVDQJRVDPDE5O5VUDQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567347221 (573h6m56.438430256s) [noise=true , address=145.79.227.5:4222 , piece_id=SEYX67D3UBXQCBDT3XOHRSWGNPBU63GT3ZNSC7XUKXA7NVKVBGXA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.567788782 (573h6m56.438871817s) [noise=true , address=145.79.227.17:4971 , piece_id=3JD35EOECXSTO3BMZ3XTCICMBIRVFZUTPQLKJMUVOCYU72IPI75A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568135863 (573h6m56.439218908s) [noise=true , address=145.79.227.45:4231 , piece_id=GCOXLBWZNZPQWLOTDQ4U3XWPZR2XMKBIOWZT2VYBG2QLVTRD6SFQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568463307 (573h6m56.439546342s) [noise=true , address=145.79.227.52:4446 , piece_id=UHTHU6UCDVXFEMUO57CPRBN45FZ4VSCZMRF4VUHK6Q54ZFWYVGJQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.568793405 (573h6m56.43987644s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.571479169 (573h6m56.442562214s) [noise=true , address=145.79.227.15:4873 , piece_id=HXJ7MUK5NXVDGK475QLSS2SWBBBYDU2E52TIJR5OPLPHZI73IC7A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.572622166 (573h6m56.443705201s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDGD5Z4FJVTIP5AXTB4R6ATEQ5YKU6HE4B4YRLAKWNN2CGDMAUA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.573898686 (573h6m56.444981721s) [noise=true , address=145.79.227.4:4187 , piece_id=O7DY3KACKXB6Z3ZZFMN2W3JWL7R7XUIMZDF3O5TGMBDHA6ASP3HA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.574956531 (573h6m56.446039566s) [noise=true , address=145.79.227.48:4302 , piece_id=FMNLGFHK77LWP3XIPQOGY4K43UFDCFO3YJQLLENXN7DR4UCWUE4A, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.575308882 (573h6m56.446391917s) [noise=true , address=45.91.116.14:4191 , piece_id=TOACUKBN4S2P5BBEUYA5RQSOLFBLZW2OUGHXTHE54X6BYRQQ2CEQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.576339433 (573h6m56.447422468s) [noise=true , address=145.79.227.19:4078 , piece_id=7RMN7V72KNQJ5XQOY6A55FKVBLYRESRMRQDDQQ2CW4W57BPW7EYA, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.576879522 (573h6m56.447962557s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.577692750 (573h6m56.448775795s) [noise=true , address=145.79.227.42:4122 , piece_id=TYWKGHSQS767HBDZLFMLBH6T7WK2VZYJJOL43NWOAYG7MKPASSUQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:56.578460442 (573h6m56.449543477s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Uploading piece... Dec 29 12:35:58.273077944 (573h6m58.144160979s) [upload=502439]: Next returned nil segment Dec 29 12:35:58.273107380 (573h6m58.144190425s) [upload=502439]: Waiting for error group managing segments... Dec 29 12:35:58.552982213 (573h6m58.424065248s) [noise=true , address=145.79.227.17:4971 , piece_id=3JD35EOECXSTO3BMZ3XTCICMBIRVFZUTPQLKJMUVOCYU72IPI75A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.553037217 (573h6m58.424120252s) [noise=true , address=145.79.227.5:4222 , piece_id=SEYX67D3UBXQCBDT3XOHRSWGNPBU63GT3ZNSC7XUKXA7NVKVBGXA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.558023892 (573h6m58.429106927s) [noise=true , address=145.79.227.4:4187 , piece_id=O7DY3KACKXB6Z3ZZFMN2W3JWL7R7XUIMZDF3O5TGMBDHA6ASP3HA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.558358510 (573h6m58.429441545s) [noise=true , address=145.79.227.36:4518 , piece_id=755BMSS3R3L65EKO3VGQRYJHLEFYCGXT27YVJCAHUJRIEPEPPJVA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.561949057 (573h6m58.433032092s) [noise=true , address=145.79.227.45:4231 , piece_id=GCOXLBWZNZPQWLOTDQ4U3XWPZR2XMKBIOWZT2VYBG2QLVTRD6SFQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.562142565 (573h6m58.4332256s) [noise=true , address=145.79.227.19:4078 , piece_id=7RMN7V72KNQJ5XQOY6A55FKVBLYRESRMRQDDQQ2CW4W57BPW7EYA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.562620154 (573h6m58.433703189s) [noise=true , address=145.79.227.42:4122 , piece_id=TYWKGHSQS767HBDZLFMLBH6T7WK2VZYJJOL43NWOAYG7MKPASSUQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.573164152 (573h6m58.444247197s) [noise=true , address=145.79.227.3:4150 , piece_id=C2SKGSU7DBMKEGNW6SQ7ESNITDHVTNGHE4JVDQJRVDPDE5O5VUDQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.574742457 (573h6m58.445825502s) [noise=true , address=145.79.227.52:4446 , piece_id=UHTHU6UCDVXFEMUO57CPRBN45FZ4VSCZMRF4VUHK6Q54ZFWYVGJQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.578854627 (573h6m58.449937672s) [noise=true , address=145.79.227.15:4873 , piece_id=HXJ7MUK5NXVDGK475QLSS2SWBBBYDU2E52TIJR5OPLPHZI73IC7A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.580262918 (573h6m58.451345953s) [noise=true , address=145.79.227.48:4302 , piece_id=FMNLGFHK77LWP3XIPQOGY4K43UFDCFO3YJQLLENXN7DR4UCWUE4A, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:58.876595519 (573h6m58.747678554s) [noise=true , address=77.90.31.18:4805 , piece_id=ETDGD5Z4FJVTIP5AXTB4R6ATEQ5YKU6HE4B4YRLAKWNN2CGDMAUA, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:59.876081688 (573h6m59.747164723s) [noise=true , address=45.91.116.14:4191 , piece_id=TOACUKBN4S2P5BBEUYA5RQSOLFBLZW2OUGHXTHE54X6BYRQQ2CEQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: <nil> Dec 29 12:35:59.876099331 (573h6m59.747182377s) [seg_pos={0 0}, upload=502439]: Segment reached optimal threshold of 13 pieces. Dec 29 12:35:59.876274745 (573h6m59.747357811s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34228->45.91.116.12:4225: use of closed network connection Dec 29 12:35:59.876285737 (573h6m59.747368782s) [noise=true , address=45.91.116.12:4225 , piece_id=4U5XB27V6GOJV2G2QZ24ZBPE5HLVAD7PZCLZCA3VCI74F6G4XEXQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876428148 (573h6m59.747511183s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:59.876433899 (573h6m59.747516934s) [noise=true , address=45.91.116.13:4154 , piece_id=QYH7DSEKKM7F7P6PJ3EUZUIH5BBFTBPW7MZ75ZNVOHWADFNCKURQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876543237 (573h6m59.747626272s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:52370->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:35:59.876552224 (573h6m59.747635269s) [noise=true , address=45.91.116.4:4937 , piece_id=G5EVO4HSZMQVIYA6JB7MSFL6QN6XCCLP55VTTVZG26OLVPS2KZLQ, seg_pos={0 0}, upload=502439]: Long tail cancellation due to optimal threshold Dec 29 12:35:59.876615244 (573h6m59.747698279s) [seg_pos={0 0}, upload=502439]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:35:59.876698392 (573h6m59.747781427s) [upload=502439]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:35:59.876703523 (573h6m59.747786558s) [upload=502439]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:36:00.186922785 (573h7m0.05800582s) [upload=502439]: Done uploading segments... Dec 29 12:36:00.186947212 (573h7m0.058030247s) [upload=502439]: Upload finished. err: <nil> Duration: 30.003094903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.......X..........X.........] duration: 45.577635294s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 45.577651515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX.] duration: 3.706965248s trace: https://jaeger.dev.storj.io/trace/4fd775f7e1cc83f9 FAIL (2025-12-29 12:35:58 UTC) orbiter warning: context deadline exceeded Duration: 3.706991269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....XX] duration: 30.002936319s trace: https://jaeger.dev.storj.io/trace/4e35e671575b849 FAIL (2025-12-29 12:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ) context deadline exceeded ======LOGS====== Dec 29 12:35:24.044815582 (573h6m23.915898617s) [upload=502429]: Starting upload Dec 29 12:35:24.044886276 (573h6m23.915969311s) [upload=502429]: Done starting upload Dec 29 12:35:24.044948275 (573h6m23.91603131s) [upload=502429]: Uploading segments... Dec 29 12:35:24.045084364 (573h6m23.916167409s) [upload=502429]: Got next segment. Inline: false Dec 29 12:35:24.045121955 (573h6m23.91620499s) [upload=502429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:35:24.045126454 (573h6m23.916209489s) [upload=502429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:35:24.287483216 (573h6m24.158566261s) [seg_pos={0 0}, upload=502429]: Begin upload segment... Dec 29 12:35:24.287536046 (573h6m24.158619081s) [seg_pos={0 0}, upload=502429]: Done begin upload segment. Dec 29 12:35:24.287562727 (573h6m24.158645762s) [noise=true , address=77.90.31.13:4627 , piece_id=N7BDRLSYN6TTRHFMTGXUOWIZTILU6YYF5GZEAQILASDFCGB3CXDQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.287675902 (573h6m24.158758947s) [noise=true , address=145.79.227.32:4834 , piece_id=WCIVXR772TG6LYLFTHENKG46ZPMTW6IUOGVISZOJ4F7L6RFRRTQA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288122433 (573h6m24.159205468s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288420340 (573h6m24.159503385s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.288803800 (573h6m24.159886835s) [noise=true , address=45.91.116.13:4153 , piece_id=IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.289091979 (573h6m24.160175024s) [noise=true , address=145.79.227.5:4222 , piece_id=DDKILONXB4HFKHCWL26DP6GBBD4A4I3ONZXUARL3RL3L3CFV2NRQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.289542066 (573h6m24.160625101s) [noise=true , address=145.79.227.26:4655 , piece_id=65YEWVC7SHYKE2P3PGPZYJDSN7FWAJT56IYA7IFYAGZ5EBCZJU7Q, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.290075622 (573h6m24.161158677s) [noise=true , address=45.91.116.14:4190 , piece_id=B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.290435267 (573h6m24.161518312s) [noise=true , address=145.79.227.3:4149 , piece_id=DZIBDRTWGBOXZYMCVMNITJHFZSG27TYDYT4HQTGNBLPKHKL5GJPQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.291668425 (573h6m24.16275148s) [noise=true , address=145.79.227.4:4185 , piece_id=E36ZVT2O4M65WONJZRKQRSU7HBAZQNRSDHONXMVEM3L62YGTW3PA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.292660214 (573h6m24.163743259s) [noise=true , address=145.79.227.42:4121 , piece_id=5WIFOHDUBGAARBHJ2XKG2ETNB45YQ6VEBMAMHM7EKDAAB3PNV4MQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.294086660 (573h6m24.165169695s) [noise=true , address=145.79.227.21:4473 , piece_id=WVW7EH3IZGQBQLVCHOHENLZRZP4QGO3ZWHDPNYNAAOYONI44J7CA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.295434696 (573h6m24.166517731s) [noise=true , address=145.79.227.50:4374 , piece_id=QTNBBE7B3KOVMZG4CFQZWTYBLI4HFAJIZEMLKV7JSG74WNTW7IOQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.296744150 (573h6m24.167827195s) [noise=true , address=145.79.227.28:4725 , piece_id=OVRZCIYCFACNX4XQKLOHMTJKP6GHLTELIZCA6JF5Z7RKPXABQCDQ, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.298077228 (573h6m24.169160263s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:24.299128450 (573h6m24.170211485s) [noise=true , address=145.79.227.38:4978 , piece_id=P3EBFDHECSKYXUPXMTQA54A7GH3EBM26SMKCU7MJR2MVDAECWVSA, seg_pos={0 0}, upload=502429]: Uploading piece... Dec 29 12:35:25.918110760 (573h6m25.789193806s) [upload=502429]: Next returned nil segment Dec 29 12:35:25.918124516 (573h6m25.789207561s) [upload=502429]: Waiting for error group managing segments... Dec 29 12:35:26.274366734 (573h6m26.145449769s) [noise=true , address=145.79.227.5:4222 , piece_id=DDKILONXB4HFKHCWL26DP6GBBD4A4I3ONZXUARL3RL3L3CFV2NRQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.274616219 (573h6m26.145699264s) [noise=true , address=145.79.227.4:4185 , piece_id=E36ZVT2O4M65WONJZRKQRSU7HBAZQNRSDHONXMVEM3L62YGTW3PA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.278366591 (573h6m26.149449636s) [noise=true , address=145.79.227.3:4149 , piece_id=DZIBDRTWGBOXZYMCVMNITJHFZSG27TYDYT4HQTGNBLPKHKL5GJPQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.279706392 (573h6m26.150789427s) [noise=true , address=145.79.227.50:4374 , piece_id=QTNBBE7B3KOVMZG4CFQZWTYBLI4HFAJIZEMLKV7JSG74WNTW7IOQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.284638052 (573h6m26.155721087s) [noise=true , address=145.79.227.32:4834 , piece_id=WCIVXR772TG6LYLFTHENKG46ZPMTW6IUOGVISZOJ4F7L6RFRRTQA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.285181236 (573h6m26.156264291s) [noise=true , address=145.79.227.28:4725 , piece_id=OVRZCIYCFACNX4XQKLOHMTJKP6GHLTELIZCA6JF5Z7RKPXABQCDQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.301041737 (573h6m26.172124772s) [noise=true , address=145.79.227.21:4473 , piece_id=WVW7EH3IZGQBQLVCHOHENLZRZP4QGO3ZWHDPNYNAAOYONI44J7CA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.301536389 (573h6m26.172619434s) [noise=true , address=145.79.227.38:4978 , piece_id=P3EBFDHECSKYXUPXMTQA54A7GH3EBM26SMKCU7MJR2MVDAECWVSA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.306014897 (573h6m26.177097942s) [noise=true , address=145.79.227.42:4121 , piece_id=5WIFOHDUBGAARBHJ2XKG2ETNB45YQ6VEBMAMHM7EKDAAB3PNV4MQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.306836561 (573h6m26.177919596s) [noise=true , address=145.79.227.26:4655 , piece_id=65YEWVC7SHYKE2P3PGPZYJDSN7FWAJT56IYA7IFYAGZ5EBCZJU7Q, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.556961686 (573h6m26.428044721s) [noise=true , address=45.91.116.13:4153 , piece_id=IMCJGAJ2ODHJP2FZOSDPHMBEGIG6GJ37L6U5Q7U3IC6PDXBTUCWQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.565925003 (573h6m26.437008038s) [noise=true , address=45.91.116.14:4190 , piece_id=B4BO3I4FPLZ7FTVHJFSVGT6SDG3IUJYP5QIGXZZCAFOI6DNBHPYA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.610410009 (573h6m26.481493044s) [noise=true , address=77.90.31.13:4627 , piece_id=N7BDRLSYN6TTRHFMTGXUOWIZTILU6YYF5GZEAQILASDFCGB3CXDQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: <nil> Dec 29 12:35:26.610427303 (573h6m26.481510338s) [seg_pos={0 0}, upload=502429]: Segment reached optimal threshold of 13 pieces. Dec 29 12:35:26.610672970 (573h6m26.481756016s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:26.610683330 (573h6m26.481766365s) [noise=true , address=145.79.227.49:4338 , piece_id=M3FAA24AAD5SN2WBDRKPWMQLFLRCQBPSUVCSETC3YCIB55EHYCLQ, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.610952302 (573h6m26.482035337s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35696->77.90.31.11:4553: use of closed network connection Dec 29 12:35:26.610962682 (573h6m26.482045737s) [noise=true , address=77.90.31.11:4553 , piece_id=M5UP67N6S5A4QAZ7ESCASC3RYZYXKIUMX3CFJKKTOPMSU6ZDUQBA, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.610982750 (573h6m26.482065785s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 12:35:26.611001165 (573h6m26.4820842s) [noise=true , address=145.79.227.35:4482 , piece_id=FLFKAN6YIDLTZAAH6J4A4AJFDPJMJIUKRM566ZNWWRDGREDLXMMQ, seg_pos={0 0}, upload=502429]: Long tail cancellation due to optimal threshold Dec 29 12:35:26.611060909 (573h6m26.482143944s) [seg_pos={0 0}, upload=502429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:35:26.611127947 (573h6m26.482210982s) [upload=502429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:35:26.611155329 (573h6m26.482238364s) [upload=502429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:35:26.918139471 (573h6m26.789222516s) [upload=502429]: Done uploading segments... Dec 29 12:35:26.918164068 (573h6m26.789247103s) [upload=502429]: Upload finished. err: <nil> Duration: 30.002956148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.......X..........X........] duration: 42.088652307s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.088669319s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....X] duration: 30.002823037s trace: https://jaeger.dev.storj.io/trace/29290dcdebfdabb FAIL (2025-12-29 12:35:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ) context deadline exceeded ======LOGS====== m42.937291868s) [upload=502419]: Uploading segments... Dec 29 12:34:43.066323461 (573h5m42.937406496s) [upload=502419]: Got next segment. Inline: false Dec 29 12:34:43.066369318 (573h5m42.937452353s) [upload=502419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:34:43.066373927 (573h5m42.937456962s) [upload=502419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:34:43.325012811 (573h5m43.196095856s) [seg_pos={0 0}, upload=502419]: Begin upload segment... Dec 29 12:34:43.325067394 (573h5m43.196150429s) [seg_pos={0 0}, upload=502419]: Done begin upload segment. Dec 29 12:34:43.325084688 (573h5m43.196167723s) [noise=true , address=145.79.227.20:4437 , piece_id=P4ORQRTHDIZT3MKDLIKJJNJKNTZBOIKN4BGB46B5F2WI6GR4XIUA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325097221 (573h5m43.196180256s) [noise=true , address=77.90.31.14:4661 , piece_id=OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325255443 (573h5m43.196338478s) [noise=true , address=145.79.227.47:4267 , piece_id=IDV6KZCBTCKY2JM2U5B2GYRFSDGQDIK33JJFAMWCKUFYBPZIRTSA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.325549322 (573h5m43.196632357s) [noise=true , address=145.79.227.21:4473 , piece_id=UYGLGKEX7MXSPSVZ5SQU42WLSHIU2FPI4HHJUK63EYTYRI2KTZ6A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326029567 (573h5m43.197112602s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326298168 (573h5m43.197381204s) [noise=true , address=77.90.31.21:4878 , piece_id=5NZHBG4LBZU74LGWGEAT5IOF37EMVDQAE7ZVPZSWD2E3IRMV4G7A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326545079 (573h5m43.197628114s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.326827136 (573h5m43.197910171s) [noise=true , address=145.79.227.41:4086 , piece_id=BBE4OMYOUHHRA5LKZ4ACYELN3AUVPDIA7PKBSTESTWN3YO2GDUFQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327137436 (573h5m43.198220502s) [noise=true , address=145.79.227.5:4221 , piece_id=S43XISH5LCQDSKRSRHNBX4I7WM2IRYYSPWXLKPJVAFQKOEL2776A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327510146 (573h5m43.198593182s) [noise=true , address=145.79.227.3:4149 , piece_id=4PCA6YHFKOTB5P5X4YWFELQ4W3OUHWCSDYDB5FBO2NKBZIIWLDEQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.327834744 (573h5m43.198917779s) [noise=true , address=145.79.227.38:4977 , piece_id=7KLAXLYXGYNXBMTVHAGRP2XZS7MF2UJ42367L6CHWZIAXBQANNZA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.328725660 (573h5m43.199808695s) [noise=true , address=77.90.31.16:4735 , piece_id=CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.328838596 (573h5m43.199921641s) [noise=true , address=145.79.227.16:4909 , piece_id=U37QA2QKVMAE4P55CVYHXL2XEM3P3IFM2X24J25PL3WDDLVRXCWA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.329920797 (573h5m43.201003832s) [noise=true , address=145.79.227.39:4013 , piece_id=DXWQD2L57XBL2QNPN2BV3YXEAY7JQJJMFQXCLKPMB2RKZ74JJYTQ, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.331052321 (573h5m43.202135356s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:43.332046695 (573h5m43.20312973s) [noise=true , address=145.79.227.26:4654 , piece_id=A7BOQWYJQRIMLU7UAQ46NAWRWQ3SIYAV3YQIHK5DYAD4VGRDBY2A, seg_pos={0 0}, upload=502419]: Uploading piece... Dec 29 12:34:44.936900204 (573h5m44.80798324s) [upload=502419]: Next returned nil segment Dec 29 12:34:44.936914050 (573h5m44.807997086s) [upload=502419]: Waiting for error group managing segments... Dec 29 12:34:45.313615295 (573h5m45.18469833s) [noise=true , address=145.79.227.21:4473 , piece_id=UYGLGKEX7MXSPSVZ5SQU42WLSHIU2FPI4HHJUK63EYTYRI2KTZ6A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.314226138 (573h5m45.185309183s) [noise=true , address=145.79.227.20:4437 , piece_id=P4ORQRTHDIZT3MKDLIKJJNJKNTZBOIKN4BGB46B5F2WI6GR4XIUA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.315203449 (573h5m45.186286484s) [noise=true , address=145.79.227.41:4086 , piece_id=BBE4OMYOUHHRA5LKZ4ACYELN3AUVPDIA7PKBSTESTWN3YO2GDUFQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.317324057 (573h5m45.188407102s) [noise=true , address=145.79.227.5:4221 , piece_id=S43XISH5LCQDSKRSRHNBX4I7WM2IRYYSPWXLKPJVAFQKOEL2776A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.318854060 (573h5m45.189937105s) [noise=true , address=145.79.227.16:4909 , piece_id=U37QA2QKVMAE4P55CVYHXL2XEM3P3IFM2X24J25PL3WDDLVRXCWA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.319441118 (573h5m45.190524153s) [noise=true , address=145.79.227.3:4149 , piece_id=4PCA6YHFKOTB5P5X4YWFELQ4W3OUHWCSDYDB5FBO2NKBZIIWLDEQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.320131413 (573h5m45.191214458s) [noise=true , address=145.79.227.38:4977 , piece_id=7KLAXLYXGYNXBMTVHAGRP2XZS7MF2UJ42367L6CHWZIAXBQANNZA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.325024109 (573h5m45.196107144s) [noise=true , address=145.79.227.26:4654 , piece_id=A7BOQWYJQRIMLU7UAQ46NAWRWQ3SIYAV3YQIHK5DYAD4VGRDBY2A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.327497859 (573h5m45.198580894s) [noise=true , address=145.79.227.47:4267 , piece_id=IDV6KZCBTCKY2JM2U5B2GYRFSDGQDIK33JJFAMWCKUFYBPZIRTSA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.346052220 (573h5m45.217135255s) [noise=true , address=145.79.227.39:4013 , piece_id=DXWQD2L57XBL2QNPN2BV3YXEAY7JQJJMFQXCLKPMB2RKZ74JJYTQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.521418984 (573h5m45.392502029s) [noise=true , address=77.90.31.21:4878 , piece_id=5NZHBG4LBZU74LGWGEAT5IOF37EMVDQAE7ZVPZSWD2E3IRMV4G7A, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.620408424 (573h5m45.491491469s) [noise=true , address=77.90.31.16:4735 , piece_id=CXJEXDSXFCZBBBF447GL6INCXX5VX6M2EXGMFW64XQBJWDTGGUHQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.645376457 (573h5m45.516459502s) [noise=true , address=77.90.31.14:4661 , piece_id=OAXXRPE2HCSYT6ECJJN3QCTW7RX6RQZGMDAMZDYAR7PDQ4IBPYJA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: <nil> Dec 29 12:34:45.645398809 (573h5m45.516481854s) [seg_pos={0 0}, upload=502419]: Segment reached optimal threshold of 13 pieces. Dec 29 12:34:45.645677029 (573h5m45.516760064s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41234->45.91.116.12:4226: use of closed network connection Dec 29 12:34:45.645704241 (573h5m45.516787276s) [noise=true , address=45.91.116.12:4226 , piece_id=JMU7N3SBOBGTKSP35UQF3EUC3OXGN4F4CZHUBBFLJ4HPER7BYSDQ, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.645895356 (573h5m45.516978391s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42122->45.91.116.4:4940: use of closed network connection Dec 29 12:34:45.645924140 (573h5m45.517007185s) [noise=true , address=45.91.116.4:4940 , piece_id=TWLYKO7OVPB56J5QBOSDFXHCVIWBD7FLILLYN5XX35BUV5F2HUQA, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.645956301 (573h5m45.517039336s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:34:45.645986929 (573h5m45.517069964s) [noise=true , address=45.91.116.5:4297 , piece_id=5FADDZAHAYDWDNYGXH6IS2VHLLBE4UVWUD4DCCAIJDMNTAIVSM7Q, seg_pos={0 0}, upload=502419]: Long tail cancellation due to optimal threshold Dec 29 12:34:45.646010334 (573h5m45.517093379s) [seg_pos={0 0}, upload=502419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:34:45.646113591 (573h5m45.517196626s) [upload=502419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:34:45.646118430 (573h5m45.517201465s) [upload=502419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:34:45.935447816 (573h5m45.806530861s) [upload=502419]: Done uploading segments... Dec 29 12:34:45.935475378 (573h5m45.806558413s) [upload=502419]: Upload finished. err: <nil> Duration: 30.00284547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [X...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXXX] duration: 5.001603875s trace: https://jaeger.dev.storj.io/trace/4fd775f7e1cc83f9 FAIL (2025-12-29 12:35:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001630946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.......X..........X.......] duration: 44.805290832s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 44.805307845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X....] duration: 6.012061916s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:34:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 6.01210074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXXX] duration: 5.001439456s trace: https://jaeger.dev.storj.io/trace/810ca148700c57 FAIL (2025-12-29 12:34:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001461187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.......X..........X......] duration: 42.908403656s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:34:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.908433502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXXX] duration: 5.00182055s trace: https://jaeger.dev.storj.io/trace/1adb64ac4b59522 FAIL (2025-12-29 12:33:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001850756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.......X..........X.....] duration: 42.12777906s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 42.127792847s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X...] duration: 5.949110859s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:33:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 5.949145395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXXX] duration: 5.001698468s trace: https://jaeger.dev.storj.io/trace/efa624aa50d309a FAIL (2025-12-29 12:33:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001724739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.......X..........X....] duration: 33.756777151s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:32:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 33.756803942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X..] duration: 5.94581261s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 5.94583946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [..XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXXX] duration: 5.00083816s trace: https://jaeger.dev.storj.io/trace/aee26b416e7073d FAIL (2025-12-29 12:32:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000864159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.......X..........X...] duration: 49.45006109s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 49.450080016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXXX] duration: 5.001242529s trace: https://jaeger.dev.storj.io/trace/39f499f98da83ad6 FAIL (2025-12-29 12:31:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001259361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X.] duration: 5.770530615s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:31:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 5.770558809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..X] duration: 30.005533785s trace: https://jaeger.dev.storj.io/trace/55af87306b44a0fb FAIL (2025-12-29 12:30:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:30:27.976557996 (573h1m27.847641031s) [upload=502356]: Starting upload Dec 29 12:30:27.976643769 (573h1m27.847726804s) [upload=502356]: Done starting upload Dec 29 12:30:27.976805467 (573h1m27.847888512s) [upload=502356]: Uploading segments... Dec 29 12:30:27.976959821 (573h1m27.848042856s) [upload=502356]: Got next segment. Inline: false Dec 29 12:30:27.977030205 (573h1m27.84811324s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:30:27.977037499 (573h1m27.848120544s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:30:28.227612220 (573h1m28.098695265s) [seg_pos={0 0}, upload=502356]: Begin upload segment... Dec 29 12:30:28.227680119 (573h1m28.098763154s) [seg_pos={0 0}, upload=502356]: Done begin upload segment. Dec 29 12:30:28.227731227 (573h1m28.098814272s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.227777565 (573h1m28.0988606s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228331690 (573h1m28.099414725s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228647301 (573h1m28.099730336s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.228978120 (573h1m28.100061155s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229164997 (573h1m28.100248032s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.229770089 (573h1m28.100853124s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230196019 (573h1m28.101279054s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230229163 (573h1m28.101312198s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230721130 (573h1m28.101804165s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.230975274 (573h1m28.102058309s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231314189 (573h1m28.102397235s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231509651 (573h1m28.102592686s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.231966481 (573h1m28.103049526s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232489376 (573h1m28.103572411s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:28.232858590 (573h1m28.103941625s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Uploading piece... Dec 29 12:30:29.934302048 (573h1m29.805385083s) [upload=502356]: Next returned nil segment Dec 29 12:30:29.934324049 (573h1m29.805407094s) [upload=502356]: Waiting for error group managing segments... Dec 29 12:30:30.219330104 (573h1m30.090413149s) [noise=true , address=145.79.227.2:4115 , piece_id=FYK6H2CTV3YRNPU3TW5ICDUUHNJLUGYB7YVL242N7EKXA3OFMUHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.222585522 (573h1m30.093668557s) [noise=true , address=145.79.227.26:4654 , piece_id=CJJW6NM2HL572SDZSTGZID6S6PRCFPS6DEJMQU4WN3DX2GMMLEHQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.223473883 (573h1m30.094556928s) [noise=true , address=145.79.227.7:4293 , piece_id=6FYYAD6K52LDUNKOGFFAGNLPFHKA7V3WVBRPKHOZBZXMHRHIACYA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.225048181 (573h1m30.096131226s) [noise=true , address=145.79.227.32:4835 , piece_id=GI4T5K7YGHDDF75UUH722QEZYSYU3DRXS4HPYWLLADM6J2EHNR5A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.227668892 (573h1m30.098751937s) [noise=true , address=145.79.227.10:4401 , piece_id=TCQXBURRVQY5VJBL6CVM56AVQ23IALWXMJYCXI2FQNK5VEA7ZP6A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.228574516 (573h1m30.099657561s) [noise=true , address=145.79.227.13:4802 , piece_id=MR6H5CX267HPWCETWENONX5QMETKXN3SL5LKWYNBNDEMBWR46ZAA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.234122320 (573h1m30.105205355s) [noise=true , address=145.79.227.45:4231 , piece_id=AI4FTKKS7SWUMAQ35E4ML7TT3O2CGOZHAA5FQPIM4L2TSDOV3TSQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.240891529 (573h1m30.111974574s) [noise=true , address=145.79.227.48:4301 , piece_id=VEJVLLQU2KDKYM3Y5XZB6TNUUXSX2MDC6Y2KXN6RUEWASB7G6TAQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.241789950 (573h1m30.112872985s) [noise=true , address=145.79.227.46:4601 , piece_id=C3VY24KTEOXUNLPAVDT45J2LRQPQ6AOR2OKTEJGLARZOUGOS64YQ, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.247350498 (573h1m30.118433534s) [noise=true , address=145.79.227.41:4085 , piece_id=MZYJNEXIKOQVVEZPIEZ33W6XJWKMY3LLYKVUW5PNSRHCD7Y6AZQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.417471593 (573h1m30.288554628s) [noise=true , address=77.90.31.16:4735 , piece_id=LED3MXVURDX5ITUWGZGJHXM2LXAX3ISSJKAECUJUOABOLUHEAKKA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.432824027 (573h1m30.303907062s) [noise=true , address=45.91.116.10:4083 , piece_id=AYQXAYPZMWZN2YSVCNUENMW4KCE6BFBEBV2N57BRRDLNYL5ZCLTA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583724624 (573h1m30.454807659s) [noise=true , address=45.91.116.14:4190 , piece_id=SVSZXQDV6WXOB5BID3HVGTAPTKSOJ2PXSGGW3OIQ5HEZRGVRG6DA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: <nil> Dec 29 12:30:30.583743831 (573h1m30.454826876s) [seg_pos={0 0}, upload=502356]: Segment reached optimal threshold of 13 pieces. Dec 29 12:30:30.583922962 (573h1m30.455005997s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.583929113 (573h1m30.455012149s) [noise=true , address=45.91.116.5:4297 , piece_id=63ABPGW4YSCEPCPMQHFZ6NZVWNYJAELGJNXJJXWFTHVTW6YIVAQA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584091974 (573h1m30.455175009s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:30:30.584108805 (573h1m30.455191841s) [noise=true , address=45.91.116.6:4974 , piece_id=NCAE2UNBQ7IAI2B2EE5QKMIJQZ5NPAMBGEYODLFMERSD5IVYUUHA, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584341749 (573h1m30.455424794s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: context canceled; context canceled Dec 29 12:30:30.584349564 (573h1m30.455432609s) [noise=true , address=45.91.116.12:4227 , piece_id=34PI32NCVIDSE2W6UAATADRFY2ZKVIS5DOXTHGL2TWDNUYZK5I4A, seg_pos={0 0}, upload=502356]: Long tail cancellation due to optimal threshold Dec 29 12:30:30.584369573 (573h1m30.455452608s) [seg_pos={0 0}, upload=502356]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:30:30.584449214 (573h1m30.455532259s) [upload=502356]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:30:30.584453732 (573h1m30.455536767s) [upload=502356]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:30:30.886256078 (573h1m30.757339123s) [upload=502356]: Done uploading segments... Dec 29 12:30:30.886281006 (573h1m30.757364041s) [upload=502356]: Upload finished. err: <nil> Duration: 30.005558082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.......X..........X..] duration: 52.625956373s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:30:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 52.625973786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [X...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXXX] duration: 5.001366684s trace: https://jaeger.dev.storj.io/trace/29679e50f1de0451 FAIL (2025-12-29 12:30:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001391011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X..] duration: 6.911981917s trace: https://jaeger.dev.storj.io/trace/dce2d9c07ebeda FAIL (2025-12-29 12:30:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:29:13.346940948 (573h0m13.218023983s) [upload=502335]: Starting upload Dec 29 12:29:13.347011723 (573h0m13.218094768s) [upload=502335]: Done starting upload Dec 29 12:29:13.347109640 (573h0m13.218192675s) [upload=502335]: Uploading segments... Dec 29 12:29:13.347238846 (573h0m13.218321881s) [upload=502335]: Got next segment. Inline: false Dec 29 12:29:13.347286016 (573h0m13.218369051s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:29:13.347292307 (573h0m13.218375342s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:29:13.615185145 (573h0m13.48626821s) [seg_pos={0 0}, upload=502335]: Begin upload segment... Dec 29 12:29:13.615238265 (573h0m13.486321301s) [seg_pos={0 0}, upload=502335]: Done begin upload segment. Dec 29 12:29:13.615269796 (573h0m13.486352831s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.615443436 (573h0m13.486526481s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616491041 (573h0m13.487574086s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616837881 (573h0m13.487920917s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617296364 (573h0m13.488379409s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617643375 (573h0m13.48872641s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617839659 (573h0m13.488922704s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618093743 (573h0m13.489176778s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618256753 (573h0m13.489339799s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618478455 (573h0m13.4895615s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618713713 (573h0m13.489796759s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618933772 (573h0m13.490016817s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619144123 (573h0m13.490227158s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619427203 (573h0m13.490510248s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619559173 (573h0m13.490642208s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619965547 (573h0m13.491048592s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:15.216249353 (573h0m15.087332388s) [upload=502335]: Next returned nil segment Dec 29 12:29:15.216283378 (573h0m15.087366413s) [upload=502335]: Waiting for error group managing segments... Dec 29 12:29:15.602828155 (573h0m15.4739112s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.603399572 (573h0m15.474482608s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.612424137 (573h0m15.483507172s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.617153132 (573h0m15.488236167s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.619789702 (573h0m15.490872747s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.620193140 (573h0m15.491276185s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.625756193 (573h0m15.496839238s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627063232 (573h0m15.498146267s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627976201 (573h0m15.499059246s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.628514746 (573h0m15.499597781s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.919130091 (573h0m15.790213136s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.012809920 (573h0m15.883892966s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272699189 (573h0m16.143782234s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272717483 (573h0m16.143800518s) [seg_pos={0 0}, upload=502335]: Segment reached optimal threshold of 13 pieces. Dec 29 12:29:16.272868170 (573h0m16.143951205s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272874262 (573h0m16.143957297s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.272899089 (573h0m16.143982124s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272907444 (573h0m16.143990479s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273088450 (573h0m16.144171485s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.273094651 (573h0m16.144177686s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273114569 (573h0m16.144197604s) [seg_pos={0 0}, upload=502335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:29:16.273190043 (573h0m16.144273078s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:29:16.273194472 (573h0m16.144277517s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:29:16.582350199 (573h0m16.453433235s) [upload=502335]: Done uploading segments... Dec 29 12:29:16.582376960 (573h0m16.453460035s) [upload=502335]: Upload finished. err: <nil> Duration: 6.912012405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.......X..........X.] duration: 41.798414901s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 41.798438395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXXX] duration: 5.00164341s trace: https://jaeger.dev.storj.io/trace/67bd20ee19569f03 FAIL (2025-12-29 12:29:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001665703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X.] duration: 6.00128071s trace: https://jaeger.dev.storj.io/trace/dce2d9c07ebeda FAIL (2025-12-29 12:29:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:29:13.346940948 (573h0m13.218023983s) [upload=502335]: Starting upload Dec 29 12:29:13.347011723 (573h0m13.218094768s) [upload=502335]: Done starting upload Dec 29 12:29:13.347109640 (573h0m13.218192675s) [upload=502335]: Uploading segments... Dec 29 12:29:13.347238846 (573h0m13.218321881s) [upload=502335]: Got next segment. Inline: false Dec 29 12:29:13.347286016 (573h0m13.218369051s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:29:13.347292307 (573h0m13.218375342s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:29:13.615185145 (573h0m13.48626821s) [seg_pos={0 0}, upload=502335]: Begin upload segment... Dec 29 12:29:13.615238265 (573h0m13.486321301s) [seg_pos={0 0}, upload=502335]: Done begin upload segment. Dec 29 12:29:13.615269796 (573h0m13.486352831s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.615443436 (573h0m13.486526481s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616491041 (573h0m13.487574086s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616837881 (573h0m13.487920917s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617296364 (573h0m13.488379409s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617643375 (573h0m13.48872641s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617839659 (573h0m13.488922704s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618093743 (573h0m13.489176778s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618256753 (573h0m13.489339799s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618478455 (573h0m13.4895615s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618713713 (573h0m13.489796759s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618933772 (573h0m13.490016817s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619144123 (573h0m13.490227158s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619427203 (573h0m13.490510248s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619559173 (573h0m13.490642208s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619965547 (573h0m13.491048592s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:15.216249353 (573h0m15.087332388s) [upload=502335]: Next returned nil segment Dec 29 12:29:15.216283378 (573h0m15.087366413s) [upload=502335]: Waiting for error group managing segments... Dec 29 12:29:15.602828155 (573h0m15.4739112s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.603399572 (573h0m15.474482608s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.612424137 (573h0m15.483507172s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.617153132 (573h0m15.488236167s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.619789702 (573h0m15.490872747s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.620193140 (573h0m15.491276185s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.625756193 (573h0m15.496839238s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627063232 (573h0m15.498146267s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627976201 (573h0m15.499059246s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.628514746 (573h0m15.499597781s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.919130091 (573h0m15.790213136s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.012809920 (573h0m15.883892966s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272699189 (573h0m16.143782234s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272717483 (573h0m16.143800518s) [seg_pos={0 0}, upload=502335]: Segment reached optimal threshold of 13 pieces. Dec 29 12:29:16.272868170 (573h0m16.143951205s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272874262 (573h0m16.143957297s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.272899089 (573h0m16.143982124s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272907444 (573h0m16.143990479s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273088450 (573h0m16.144171485s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.273094651 (573h0m16.144177686s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273114569 (573h0m16.144197604s) [seg_pos={0 0}, upload=502335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:29:16.273190043 (573h0m16.144273078s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:29:16.273194472 (573h0m16.144277517s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:29:16.582350199 (573h0m16.453433235s) [upload=502335]: Done uploading segments... Dec 29 12:29:16.582376960 (573h0m16.453460035s) [upload=502335]: Upload finished. err: <nil> Duration: 6.001294135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.X.X....XXXXXX..X.X.......X.XX...X.......X.X..XXX....X...X] duration: 30.004059393s trace: https://jaeger.dev.storj.io/trace/dce2d9c07ebeda FAIL (2025-12-29 12:29:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 12:29:13.346940948 (573h0m13.218023983s) [upload=502335]: Starting upload Dec 29 12:29:13.347011723 (573h0m13.218094768s) [upload=502335]: Done starting upload Dec 29 12:29:13.347109640 (573h0m13.218192675s) [upload=502335]: Uploading segments... Dec 29 12:29:13.347238846 (573h0m13.218321881s) [upload=502335]: Got next segment. Inline: false Dec 29 12:29:13.347286016 (573h0m13.218369051s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 12:29:13.347292307 (573h0m13.218375342s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 12:29:13.615185145 (573h0m13.48626821s) [seg_pos={0 0}, upload=502335]: Begin upload segment... Dec 29 12:29:13.615238265 (573h0m13.486321301s) [seg_pos={0 0}, upload=502335]: Done begin upload segment. Dec 29 12:29:13.615269796 (573h0m13.486352831s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.615443436 (573h0m13.486526481s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616491041 (573h0m13.487574086s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.616837881 (573h0m13.487920917s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617296364 (573h0m13.488379409s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617643375 (573h0m13.48872641s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.617839659 (573h0m13.488922704s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618093743 (573h0m13.489176778s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618256753 (573h0m13.489339799s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618478455 (573h0m13.4895615s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618713713 (573h0m13.489796759s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.618933772 (573h0m13.490016817s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619144123 (573h0m13.490227158s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619427203 (573h0m13.490510248s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619559173 (573h0m13.490642208s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:13.619965547 (573h0m13.491048592s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Uploading piece... Dec 29 12:29:15.216249353 (573h0m15.087332388s) [upload=502335]: Next returned nil segment Dec 29 12:29:15.216283378 (573h0m15.087366413s) [upload=502335]: Waiting for error group managing segments... Dec 29 12:29:15.602828155 (573h0m15.4739112s) [noise=true , address=145.79.227.16:4912 , piece_id=55RKZVL4OXEOP3S7K7GTKKINENABN5V73LO34ODYIUWFLH6Q7EPA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.603399572 (573h0m15.474482608s) [noise=true , address=145.79.227.7:4293 , piece_id=GDJDYEOFO4FGQ3SLJH7JK5DUJQD2OHL2IOKBZPPNEJB63YTM7F4Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.612424137 (573h0m15.483507172s) [noise=true , address=145.79.227.18:4007 , piece_id=FYQRHTXX6KRYN6FQTJ6GQVUJZQKL2ZAOX2JL6ELFEX67X6CNPEJA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.617153132 (573h0m15.488236167s) [noise=true , address=145.79.227.50:4374 , piece_id=Z3W7JTR65Y6N7MSITROEMJJ6NJ77IK2GKS44WMPEDP6J2TOE7HNA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.619789702 (573h0m15.490872747s) [noise=true , address=145.79.227.45:4229 , piece_id=T43GEEDD5GABJYXB64TISMBX6TSX6YEOFKZ2S2ZQRZBMKH47SX4A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.620193140 (573h0m15.491276185s) [noise=true , address=145.79.227.48:4303 , piece_id=QIONIBQIO325OCUSNOVVOI3OOVACS35JDNXEZPSYLWMHG376UD7A, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.625756193 (573h0m15.496839238s) [noise=true , address=145.79.227.3:4151 , piece_id=2XZFZKIEVPR6PWICIMWRKSEQ3WSW5RPHSEAW4BX7FR2GDDN5TGQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627063232 (573h0m15.498146267s) [noise=true , address=145.79.227.52:4446 , piece_id=NSFB7ENJNRKUGHTYJRONCFUUG4PE7ULTBZ6LWMAPZKTM6JJBFTHQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.627976201 (573h0m15.499059246s) [noise=true , address=145.79.227.28:4725 , piece_id=X65WZ5QAWLMOJMU7OZOF2T7LX45MRB4AMRGQMHH43LW55M4B2WFQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.628514746 (573h0m15.499597781s) [noise=true , address=145.79.227.37:4567 , piece_id=LS6QLDBRF22CSPBG42E4IC6T6T4GRPNQOBRDSKQBXIDGW5NA4XLA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:15.919130091 (573h0m15.790213136s) [noise=true , address=77.90.31.13:4629 , piece_id=WOM7R2YMBAT3FMJ7VCXERRX2KRGKDA2K5KHBYGZLQHWDTYAUKB2Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.012809920 (573h0m15.883892966s) [noise=true , address=45.91.116.12:4225 , piece_id=5QF5DK7QVJ67HT2GVFBIIXUMTUC6FVVD3RBVYHIVAUIH4VGVYOQA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272699189 (573h0m16.143782234s) [noise=true , address=145.79.227.17:4970 , piece_id=RU54FB4U3LR3DLMBFMPPCS6FXDU5I7QN5GF5TIFNUQK2LIRF7R7Q, seg_pos={0 0}, upload=502335]: Done uploading piece. err: <nil> Dec 29 12:29:16.272717483 (573h0m16.143800518s) [seg_pos={0 0}, upload=502335]: Segment reached optimal threshold of 13 pieces. Dec 29 12:29:16.272868170 (573h0m16.143951205s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272874262 (573h0m16.143957297s) [noise=true , address=77.90.31.17:4770 , piece_id=IPQXLKPDNNHH6W4PWEMJ5LFWB4GY6M25IRUHP72QXTCRVYX2SHEA, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.272899089 (573h0m16.143982124s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.272907444 (573h0m16.143990479s) [noise=true , address=45.91.116.8:4046 , piece_id=GWIETJORTBYROVXTIVVXIPIQT3M5UKC2CYTCKPN6BT4UDEPOM3YQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273088450 (573h0m16.144171485s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 12:29:16.273094651 (573h0m16.144177686s) [noise=true , address=145.79.227.29:4942 , piece_id=4GY7GGGZX2F43BFA7AY5IIGTR7KTDDQZYVWTQNXONJKRVHAK7ZEQ, seg_pos={0 0}, upload=502335]: Long tail cancellation due to optimal threshold Dec 29 12:29:16.273114569 (573h0m16.144197604s) [seg_pos={0 0}, upload=502335]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 12:29:16.273190043 (573h0m16.144273078s) [upload=502335]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:29:16.273194472 (573h0m16.144277517s) [upload=502335]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:29:16.582350199 (573h0m16.453433235s) [upload=502335]: Done uploading segments... Dec 29 12:29:16.582376960 (573h0m16.453460035s) [upload=502335]: Upload finished. err: <nil> Duration: 30.004080995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.......X..........X] duration: 1m0.002316081s trace: https://jaeger.dev.storj.io/trace/68764c6cb4286314 FAIL (2025-12-29 12:29:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528204662 (572h59m20.399287707s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528497489 (572h59m20.399580524s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.528793153 (572h59m20.399876198s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529083555 (572h59m20.40016659s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529391081 (572h59m20.400474126s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.529735096 (572h59m20.400818131s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530109970 (572h59m20.401193015s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530449556 (572h59m20.401532591s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.530831554 (572h59m20.401914599s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531229512 (572h59m20.402312557s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.531698945 (572h59m20.402781991s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532204238 (572h59m20.403287283s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.532744366 (572h59m20.403827401s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533231203 (572h59m20.404314248s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:20.533740113 (572h59m20.404823148s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Uploading piece... Dec 29 12:28:21.876539279 (572h59m21.747622324s) [upload=502322]: Next returned nil segment Dec 29 12:28:21.876614272 (572h59m21.747697307s) [upload=502322]: Waiting for error group managing segments... Dec 29 12:28:22.383674429 (572h59m22.254757464s) [noise=true , address=145.79.227.18:4008 , piece_id=6QU2F4DYKYWGD4PMVM3XNGDXWJ3SXXTIOR7IM67YQDQG5G32FUJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389083709 (572h59m22.260166754s) [noise=true , address=145.79.227.16:4924 , piece_id=B32UCRPNI6VIWT4TQK6P5ABYFJPEMVCHRQD2Y66X3RL5NGRFESMQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.389450968 (572h59m22.260534003s) [noise=true , address=145.79.227.46:4625 , piece_id=6NEND2CKGIZLK5Y6IMM4XB6SXPSXUYSZIVZ7CZ25IKAHFFW33C5Q, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.418373151 (572h59m22.289456186s) [noise=true , address=145.79.227.4:4204 , piece_id=VZWOPBGR7ZDT4UUMP3BGE5UV2VDIODZTNVYCM6BUN762A35ZEPYA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.619432477 (572h59m22.490515552s) [noise=true , address=77.90.31.21:4884 , piece_id=QGSPLCQDOCJYDSG65NBSWQFG3UT3TZVURXSNU2VGSX67D5RGL4DQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:22.741876388 (572h59m22.612959423s) [noise=true , address=45.91.116.8:4051 , piece_id=N3CFZEL56CJ6WD7C3MWQVMA3S4H7ALEFX2SSMDMOPURIVNJ7CFAA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.133731187 (572h59m23.004814222s) [noise=true , address=77.90.31.16:4756 , piece_id=S2FL5TFVKUJS3RZLFIQM6ZARZY6U3TQRUHQMLCUUW4KXYWY4LJEQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.146041929 (572h59m23.017124964s) [noise=true , address=193.149.164.143:4697 , piece_id=HZZY3SUOEC7AIHSW6MKZRNZVSOGXSYBYSUKN7STBFTXWAOMDRYBA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.261354601 (572h59m23.132437636s) [noise=true , address=145.79.227.44:4208 , piece_id=FP7OUHNA2NWAGXGYWUHWRSYCNUYNVU7H7DU3BBSPEOJ5MMNYG7QQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.291387197 (572h59m23.162470243s) [noise=true , address=145.79.227.30:4768 , piece_id=B3DU77LGDSOKHJ5UNRKCG6DZYADDVY4XODQFWLDU4AOMCRTYEMVA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.318370928 (572h59m23.189453973s) [noise=true , address=145.79.227.43:4174 , piece_id=TNX3SCU3SYF35AF3VFD353RXKHKM5IEKKZLPUMH2OQG4ZGA4NYKA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.321236023 (572h59m23.192319058s) [noise=true , address=145.79.227.40:4065 , piece_id=KZCYZ3BTUFLMZMZ6YWTK3F3KX56TDB5HAEW6NWBIGNXV7A45C5UA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.408408673 (572h59m23.279491718s) [noise=true , address=145.79.227.34:4910 , piece_id=AFYOLH33YWULD5GMFBU4ZZXLECPWL7TGVLZVBASEQX5R4DB2WSFA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419570137 (572h59m23.290653172s) [noise=true , address=145.79.227.35:4510 , piece_id=H3ZBGCMNKQKECTFTZFIWRKSFMBJFXOX37776FYIJ44YTACP4ZBDA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: <nil> Dec 29 12:28:23.419592470 (572h59m23.290675515s) [seg_pos={0 0}, upload=502322]: Segment reached optimal threshold of 14 pieces. Dec 29 12:28:23.419837406 (572h59m23.290920441s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123jUFZTVjq6adDuvt3cco7dTvwNTQkszZX7CeajMfbeCUmEmuJ): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419845241 (572h59m23.290928276s) [noise=true , address=77.90.31.11:4569 , piece_id=U2F5MUBD6P7AJNOUDP65MDLCZAPVZXNQ2BY662227OQDV6KQ4QWQ, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.419873174 (572h59m23.290956209s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 29 12:28:23.419882021 (572h59m23.290965076s) [noise=true , address=145.79.227.49:4340 , piece_id=VU4TQFNSNLETPSIDPCHSJBLEX42PKCB7YMM6GZ2KGMXQUUTRMXJA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420073716 (572h59m23.291156751s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59846->64.49.13.169:4715: use of closed network connection Dec 29 12:28:23.420082262 (572h59m23.291165307s) [noise=true , address=64.49.13.169:4715 , piece_id=KBJROHPUEQR47655MSLQ7YGFPHLZIPDZMCTU5E2IMWKAZSXLQR6A, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420257506 (572h59m23.291340551s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x): protocol: read tcp 188.166.225.170:41270->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 12:28:23.420264649 (572h59m23.291347684s) [noise=true , address=77.90.31.18:4825 , piece_id=R4SHBTW2WYUEHKJC6V77ZJFUBI7XTAMRJ77ZFXNDKVEAWXEEW5JA, seg_pos={0 0}, upload=502322]: Long tail cancellation due to optimal threshold Dec 29 12:28:23.420281822 (572h59m23.291364857s) [seg_pos={0 0}, upload=502322]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 12:28:23.420364349 (572h59m23.291447394s) [upload=502322]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 12:28:23.420370270 (572h59m23.291453315s) [upload=502322]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 12:28:23.725225770 (572h59m23.596308815s) [upload=502322]: Done uploading segments... Dec 29 12:28:23.725255426 (572h59m23.596338481s) [upload=502322]: Upload finished. err: <nil> Duration: 1m0.002339796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXX...XXX...XXX..XX.X..X.X.X..X..X.XX...XXXXXXX.X..XXXXXXXXX] duration: 5.000788065s trace: https://jaeger.dev.storj.io/trace/20c6fddbfe81bd4d FAIL (2025-12-29 12:28:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000809857s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)