Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........X............................X..XX..X...............] duration:
39.075019397s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:52:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 39.075033784s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
13.901941617s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:51:59 UTC)
orbiter warning: context deadline exceeded
Duration: 13.901966675s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X............................X..XX..X..............] duration:
41.698860392s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:51:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 41.698878627s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
12.355962642s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:51:39 UTC)
orbiter warning: context deadline exceeded
Duration: 12.355990675s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX..] duration:
5.001275235s trace:
https://jaeger.dev.storj.io/trace/2d8479babd245c3e
FAIL (2025-12-30 15:51:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001301073s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...X.......X.X..........XX.......XX.XX..X....X.......X......] duration:
5.833313374s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:51:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 5.833360363s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [X.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX.] duration:
5.001339009s trace:
https://jaeger.dev.storj.io/trace/2d8479babd245c3e
FAIL (2025-12-30 15:51:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001362362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X............................X..XX..X.............] duration:
54.779356171s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:51:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 54.779374767s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [X...X.......X.X..........XX.......XX.XX..X....X.......X.....] duration:
5.987241699s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:50:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 5.987267738s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX] duration:
5.001511006s trace:
https://jaeger.dev.storj.io/trace/2d8479babd245c3e
FAIL (2025-12-30 15:50:49 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001528738s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
13.057770476s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:50:26 UTC)
orbiter warning: context deadline exceeded
Duration: 13.057796044s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X............................X..XX..X............] duration:
37.985251517s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:50:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 37.985281785s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.X...X.......X.X..........XX.......XX.XX..X....X.......X....] duration:
13.298073563s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:49:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 13.298100735s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.578280204s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:49:52 UTC)
orbiter warning: context deadline exceeded
Duration: 6.578306665s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXX] duration:
5.001268542s trace:
https://jaeger.dev.storj.io/trace/5080a6a899de4ea8
FAIL (2025-12-30 15:49:43 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001290765s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [............X............................X..XX..X...........] duration:
15.523931486s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:49:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 15.523961512s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X...X.......X.X..........XX.......XX.XX..X....X.......X...] duration:
6.051038791s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:49:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 6.05108539s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [X.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXX] duration:
5.001874595s trace:
https://jaeger.dev.storj.io/trace/4fa3e4f94d035b68
FAIL (2025-12-30 15:48:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001905604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
14.738761528s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:48:48 UTC)
orbiter warning: context deadline exceeded
Duration: 14.738787048s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X..X...X.......X.X..........XX.......XX.XX..X....X.......X..] duration:
5.897127116s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:48:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 5.89716166s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [X............X............................X..XX..X..........] duration:
36.479627409s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:48:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 36.479665812s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXX] duration:
5.001418046s trace:
https://jaeger.dev.storj.io/trace/5cdad4fa63502562
FAIL (2025-12-30 15:48:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00144142s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X..X...X.......X.X..........XX.......XX.XX..X....X.......X.] duration:
6.096044918s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:48:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 6.096074513s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
12.562710043s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:48:13 UTC)
orbiter warning: context deadline exceeded
Duration: 12.562740229s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X............X............................X..XX..X.........] duration:
22.214893947s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:48:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 22.214914146s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X..X...X.......X.X..........XX.......XX.XX..X....X.......X] duration:
30.003262867s trace:
https://jaeger.dev.storj.io/trace/161ad17247435e8e
FAIL (2025-12-30 15:47:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded
======LOGS======
30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload
Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload
Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments...
Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false
Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment...
Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment.
Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece...
Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment
Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments...
Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil>
Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection
Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold
Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments...
Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil>
Duration: 30.003278917s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXX] duration:
5.001812519s trace:
https://jaeger.dev.storj.io/trace/5c863c92ed85266c
FAIL (2025-12-30 15:47:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001827677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
14.203017702s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:47:30 UTC)
orbiter warning: context deadline exceeded
Duration: 14.203042951s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X............X............................X..XX..X........] duration:
43.346566412s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:47:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 43.346589915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XX] duration:
5.002068062s trace:
https://jaeger.dev.storj.io/trace/d034a5ceba469e0
FAIL (2025-12-30 15:47:20 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00209306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X..X..X...X.......X.X..........XX.......XX.XX..X....X.......] duration:
5.765058715s trace:
https://jaeger.dev.storj.io/trace/5f656054292009e3
FAIL (2025-12-30 15:46:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ) context deadline exceeded
======LOGS======
30 15:40:38.885361551 (600h11m38.756444586s) [upload=526044]: Starting upload
Dec 30 15:40:38.885477071 (600h11m38.756560106s) [upload=526044]: Done starting upload
Dec 30 15:40:38.885521665 (600h11m38.75660471s) [upload=526044]: Uploading segments...
Dec 30 15:40:38.885896900 (600h11m38.756979945s) [upload=526044]: Got next segment. Inline: false
Dec 30 15:40:38.885949150 (600h11m38.757032185s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:40:38.885956093 (600h11m38.757039128s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:40:39.132223600 (600h11m39.003306645s) [seg_pos={0 0}, upload=526044]: Begin upload segment...
Dec 30 15:40:39.132282372 (600h11m39.003365417s) [seg_pos={0 0}, upload=526044]: Done begin upload segment.
Dec 30 15:40:39.132315374 (600h11m39.00339842s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.132386800 (600h11m39.003469845s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133067908 (600h11m39.004150943s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133503447 (600h11m39.004586482s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134126213 (600h11m39.005209258s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134422187 (600h11m39.005505232s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134581449 (600h11m39.005664484s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134985038 (600h11m39.006068083s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135165492 (600h11m39.006248537s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135650876 (600h11m39.006733921s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135814318 (600h11m39.006897353s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136191376 (600h11m39.007274421s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136312766 (600h11m39.007395801s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136661771 (600h11m39.007744816s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136842966 (600h11m39.007926011s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.137156583 (600h11m39.008239618s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:40.839977389 (600h11m40.711060424s) [upload=526044]: Next returned nil segment
Dec 30 15:40:40.839997668 (600h11m40.711080703s) [upload=526044]: Waiting for error group managing segments...
Dec 30 15:40:41.124136919 (600h11m40.995219964s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.147195365 (600h11m41.01827841s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.148759674 (600h11m41.019842719s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.164387442 (600h11m41.035470487s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.213200708 (600h11m41.084283753s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.289985748 (600h11m41.161068803s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.335807599 (600h11m41.206890634s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.340383422 (600h11m41.211466467s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.390789361 (600h11m41.261872406s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.512772514 (600h11m41.383855559s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.532982778 (600h11m41.404065823s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.620303829 (600h11m41.491386874s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138912742 (600h11m42.009995787s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138937430 (600h11m42.010020475s) [seg_pos={0 0}, upload=526044]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:40:42.139125157 (600h11m42.010208202s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139132431 (600h11m42.010215466s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139155505 (600h11m42.01023854s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139164863 (600h11m42.010247908s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139311212 (600h11m42.010394267s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection
Dec 30 15:40:42.139320389 (600h11m42.010403424s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139340818 (600h11m42.010423853s) [seg_pos={0 0}, upload=526044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:40:42.139420510 (600h11m42.010503555s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:40:42.139502656 (600h11m42.010585691s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:40:42.448963161 (600h11m42.320046206s) [upload=526044]: Done uploading segments...
Dec 30 15:40:42.448994711 (600h11m42.320077756s) [upload=526044]: Upload finished. err: <nil>
Duration: 5.765084745s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
11.500507383s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:46:32 UTC)
orbiter warning: context deadline exceeded
Duration: 11.500534726s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X............X............................X..XX..X.......] duration:
20.720161931s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:46:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 20.720187941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [X.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.X] duration:
5.001080305s trace:
https://jaeger.dev.storj.io/trace/2c94c0864d44c031
FAIL (2025-12-30 15:46:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001103178s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XX..X..X...X.......X.X..........XX.......XX.XX..X....X......] duration:
26.253969729s trace:
https://jaeger.dev.storj.io/trace/5f656054292009e3
FAIL (2025-12-30 15:45:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ) context deadline exceeded
======LOGS======
30 15:40:38.885361551 (600h11m38.756444586s) [upload=526044]: Starting upload
Dec 30 15:40:38.885477071 (600h11m38.756560106s) [upload=526044]: Done starting upload
Dec 30 15:40:38.885521665 (600h11m38.75660471s) [upload=526044]: Uploading segments...
Dec 30 15:40:38.885896900 (600h11m38.756979945s) [upload=526044]: Got next segment. Inline: false
Dec 30 15:40:38.885949150 (600h11m38.757032185s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:40:38.885956093 (600h11m38.757039128s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:40:39.132223600 (600h11m39.003306645s) [seg_pos={0 0}, upload=526044]: Begin upload segment...
Dec 30 15:40:39.132282372 (600h11m39.003365417s) [seg_pos={0 0}, upload=526044]: Done begin upload segment.
Dec 30 15:40:39.132315374 (600h11m39.00339842s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.132386800 (600h11m39.003469845s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133067908 (600h11m39.004150943s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133503447 (600h11m39.004586482s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134126213 (600h11m39.005209258s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134422187 (600h11m39.005505232s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134581449 (600h11m39.005664484s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134985038 (600h11m39.006068083s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135165492 (600h11m39.006248537s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135650876 (600h11m39.006733921s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135814318 (600h11m39.006897353s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136191376 (600h11m39.007274421s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136312766 (600h11m39.007395801s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136661771 (600h11m39.007744816s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136842966 (600h11m39.007926011s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.137156583 (600h11m39.008239618s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:40.839977389 (600h11m40.711060424s) [upload=526044]: Next returned nil segment
Dec 30 15:40:40.839997668 (600h11m40.711080703s) [upload=526044]: Waiting for error group managing segments...
Dec 30 15:40:41.124136919 (600h11m40.995219964s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.147195365 (600h11m41.01827841s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.148759674 (600h11m41.019842719s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.164387442 (600h11m41.035470487s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.213200708 (600h11m41.084283753s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.289985748 (600h11m41.161068803s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.335807599 (600h11m41.206890634s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.340383422 (600h11m41.211466467s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.390789361 (600h11m41.261872406s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.512772514 (600h11m41.383855559s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.532982778 (600h11m41.404065823s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.620303829 (600h11m41.491386874s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138912742 (600h11m42.009995787s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138937430 (600h11m42.010020475s) [seg_pos={0 0}, upload=526044]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:40:42.139125157 (600h11m42.010208202s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139132431 (600h11m42.010215466s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139155505 (600h11m42.01023854s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139164863 (600h11m42.010247908s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139311212 (600h11m42.010394267s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection
Dec 30 15:40:42.139320389 (600h11m42.010403424s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139340818 (600h11m42.010423853s) [seg_pos={0 0}, upload=526044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:40:42.139420510 (600h11m42.010503555s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:40:42.139502656 (600h11m42.010585691s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:40:42.448963161 (600h11m42.320046206s) [upload=526044]: Done uploading segments...
Dec 30 15:40:42.448994711 (600h11m42.320077756s) [upload=526044]: Upload finished. err: <nil>
Duration: 26.25399617s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XX.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.] duration:
3.504133294s trace:
https://jaeger.dev.storj.io/trace/7703aa5ed4b49ed5
FAIL (2025-12-30 15:45:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.504154604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
11.805209327s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:45:23 UTC)
orbiter warning: context deadline exceeded
Duration: 11.805239636s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X............X............................X..XX..X......] duration:
17.917809371s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:45:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 17.917843686s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXX.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX] duration:
5.000872386s trace:
https://jaeger.dev.storj.io/trace/7703aa5ed4b49ed5
FAIL (2025-12-30 15:44:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000897694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.XX..X..X...X.......X.X..........XX.......XX.XX..X....X.....] duration:
6.173044816s trace:
https://jaeger.dev.storj.io/trace/5f656054292009e3
FAIL (2025-12-30 15:44:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ) context deadline exceeded
======LOGS======
30 15:40:38.885361551 (600h11m38.756444586s) [upload=526044]: Starting upload
Dec 30 15:40:38.885477071 (600h11m38.756560106s) [upload=526044]: Done starting upload
Dec 30 15:40:38.885521665 (600h11m38.75660471s) [upload=526044]: Uploading segments...
Dec 30 15:40:38.885896900 (600h11m38.756979945s) [upload=526044]: Got next segment. Inline: false
Dec 30 15:40:38.885949150 (600h11m38.757032185s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:40:38.885956093 (600h11m38.757039128s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:40:39.132223600 (600h11m39.003306645s) [seg_pos={0 0}, upload=526044]: Begin upload segment...
Dec 30 15:40:39.132282372 (600h11m39.003365417s) [seg_pos={0 0}, upload=526044]: Done begin upload segment.
Dec 30 15:40:39.132315374 (600h11m39.00339842s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.132386800 (600h11m39.003469845s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133067908 (600h11m39.004150943s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133503447 (600h11m39.004586482s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134126213 (600h11m39.005209258s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134422187 (600h11m39.005505232s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134581449 (600h11m39.005664484s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134985038 (600h11m39.006068083s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135165492 (600h11m39.006248537s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135650876 (600h11m39.006733921s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135814318 (600h11m39.006897353s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136191376 (600h11m39.007274421s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136312766 (600h11m39.007395801s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136661771 (600h11m39.007744816s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136842966 (600h11m39.007926011s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.137156583 (600h11m39.008239618s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:40.839977389 (600h11m40.711060424s) [upload=526044]: Next returned nil segment
Dec 30 15:40:40.839997668 (600h11m40.711080703s) [upload=526044]: Waiting for error group managing segments...
Dec 30 15:40:41.124136919 (600h11m40.995219964s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.147195365 (600h11m41.01827841s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.148759674 (600h11m41.019842719s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.164387442 (600h11m41.035470487s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.213200708 (600h11m41.084283753s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.289985748 (600h11m41.161068803s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.335807599 (600h11m41.206890634s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.340383422 (600h11m41.211466467s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.390789361 (600h11m41.261872406s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.512772514 (600h11m41.383855559s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.532982778 (600h11m41.404065823s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.620303829 (600h11m41.491386874s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138912742 (600h11m42.009995787s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138937430 (600h11m42.010020475s) [seg_pos={0 0}, upload=526044]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:40:42.139125157 (600h11m42.010208202s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139132431 (600h11m42.010215466s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139155505 (600h11m42.01023854s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139164863 (600h11m42.010247908s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139311212 (600h11m42.010394267s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection
Dec 30 15:40:42.139320389 (600h11m42.010403424s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139340818 (600h11m42.010423853s) [seg_pos={0 0}, upload=526044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:40:42.139420510 (600h11m42.010503555s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:40:42.139502656 (600h11m42.010585691s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:40:42.448963161 (600h11m42.320046206s) [upload=526044]: Done uploading segments...
Dec 30 15:40:42.448994711 (600h11m42.320077756s) [upload=526044]: Upload finished. err: <nil>
Duration: 6.173075233s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....X............X............................X..XX..X.....] duration:
37.889275723s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:44:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 37.889302214s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXX.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXX] duration:
5.001430352s trace:
https://jaeger.dev.storj.io/trace/3585b5db64b17e71
FAIL (2025-12-30 15:44:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001451523s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X..................................X........................] duration:
12.752075435s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:44:16 UTC)
orbiter warning: context deadline exceeded
Duration: 12.752104662s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [..XX..X..X...X.......X.X..........XX.......XX.XX..X....X....] duration:
5.900653371s trace:
https://jaeger.dev.storj.io/trace/5f656054292009e3
FAIL (2025-12-30 15:44:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ) context deadline exceeded
======LOGS======
30 15:40:38.885361551 (600h11m38.756444586s) [upload=526044]: Starting upload
Dec 30 15:40:38.885477071 (600h11m38.756560106s) [upload=526044]: Done starting upload
Dec 30 15:40:38.885521665 (600h11m38.75660471s) [upload=526044]: Uploading segments...
Dec 30 15:40:38.885896900 (600h11m38.756979945s) [upload=526044]: Got next segment. Inline: false
Dec 30 15:40:38.885949150 (600h11m38.757032185s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:40:38.885956093 (600h11m38.757039128s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:40:39.132223600 (600h11m39.003306645s) [seg_pos={0 0}, upload=526044]: Begin upload segment...
Dec 30 15:40:39.132282372 (600h11m39.003365417s) [seg_pos={0 0}, upload=526044]: Done begin upload segment.
Dec 30 15:40:39.132315374 (600h11m39.00339842s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.132386800 (600h11m39.003469845s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133067908 (600h11m39.004150943s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133503447 (600h11m39.004586482s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134126213 (600h11m39.005209258s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134422187 (600h11m39.005505232s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134581449 (600h11m39.005664484s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134985038 (600h11m39.006068083s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135165492 (600h11m39.006248537s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135650876 (600h11m39.006733921s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135814318 (600h11m39.006897353s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136191376 (600h11m39.007274421s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136312766 (600h11m39.007395801s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136661771 (600h11m39.007744816s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136842966 (600h11m39.007926011s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.137156583 (600h11m39.008239618s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:40.839977389 (600h11m40.711060424s) [upload=526044]: Next returned nil segment
Dec 30 15:40:40.839997668 (600h11m40.711080703s) [upload=526044]: Waiting for error group managing segments...
Dec 30 15:40:41.124136919 (600h11m40.995219964s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.147195365 (600h11m41.01827841s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.148759674 (600h11m41.019842719s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.164387442 (600h11m41.035470487s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.213200708 (600h11m41.084283753s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.289985748 (600h11m41.161068803s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.335807599 (600h11m41.206890634s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.340383422 (600h11m41.211466467s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.390789361 (600h11m41.261872406s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.512772514 (600h11m41.383855559s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.532982778 (600h11m41.404065823s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.620303829 (600h11m41.491386874s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138912742 (600h11m42.009995787s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138937430 (600h11m42.010020475s) [seg_pos={0 0}, upload=526044]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:40:42.139125157 (600h11m42.010208202s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139132431 (600h11m42.010215466s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139155505 (600h11m42.01023854s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139164863 (600h11m42.010247908s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139311212 (600h11m42.010394267s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection
Dec 30 15:40:42.139320389 (600h11m42.010403424s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139340818 (600h11m42.010423853s) [seg_pos={0 0}, upload=526044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:40:42.139420510 (600h11m42.010503555s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:40:42.139502656 (600h11m42.010585691s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:40:42.448963161 (600h11m42.320046206s) [upload=526044]: Done uploading segments...
Dec 30 15:40:42.448994711 (600h11m42.320077756s) [upload=526044]: Upload finished. err: <nil>
Duration: 5.900676978s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XXXXX.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXX] duration:
5.001689994s trace:
https://jaeger.dev.storj.io/trace/1a8e9baf8c995443
FAIL (2025-12-30 15:44:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001713399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [...XX..X..X...X.......X.X..........XX.......XX.XX..X....X...] duration:
22.064782992s trace:
https://jaeger.dev.storj.io/trace/5f656054292009e3
FAIL (2025-12-30 15:43:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ) context deadline exceeded
======LOGS======
30 15:40:38.885361551 (600h11m38.756444586s) [upload=526044]: Starting upload
Dec 30 15:40:38.885477071 (600h11m38.756560106s) [upload=526044]: Done starting upload
Dec 30 15:40:38.885521665 (600h11m38.75660471s) [upload=526044]: Uploading segments...
Dec 30 15:40:38.885896900 (600h11m38.756979945s) [upload=526044]: Got next segment. Inline: false
Dec 30 15:40:38.885949150 (600h11m38.757032185s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 15:40:38.885956093 (600h11m38.757039128s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 15:40:39.132223600 (600h11m39.003306645s) [seg_pos={0 0}, upload=526044]: Begin upload segment...
Dec 30 15:40:39.132282372 (600h11m39.003365417s) [seg_pos={0 0}, upload=526044]: Done begin upload segment.
Dec 30 15:40:39.132315374 (600h11m39.00339842s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.132386800 (600h11m39.003469845s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133067908 (600h11m39.004150943s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.133503447 (600h11m39.004586482s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134126213 (600h11m39.005209258s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134422187 (600h11m39.005505232s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134581449 (600h11m39.005664484s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.134985038 (600h11m39.006068083s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135165492 (600h11m39.006248537s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135650876 (600h11m39.006733921s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.135814318 (600h11m39.006897353s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136191376 (600h11m39.007274421s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136312766 (600h11m39.007395801s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136661771 (600h11m39.007744816s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.136842966 (600h11m39.007926011s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:39.137156583 (600h11m39.008239618s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Uploading piece...
Dec 30 15:40:40.839977389 (600h11m40.711060424s) [upload=526044]: Next returned nil segment
Dec 30 15:40:40.839997668 (600h11m40.711080703s) [upload=526044]: Waiting for error group managing segments...
Dec 30 15:40:41.124136919 (600h11m40.995219964s) [noise=true , address=145.79.227.11:4732 , piece_id=KMIOZ4OA6H2SCSKR4KUCCF42RMN3OTHRMOSAQSQ6Y7QGCGKZLDKA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.147195365 (600h11m41.01827841s) [noise=true , address=145.79.227.5:4222 , piece_id=BYPNOXRMGYIJPXOCOKH6YS6FR4VKJ3OVKU7MVBX5TQKNCEXWEL3Q, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.148759674 (600h11m41.019842719s) [noise=true , address=145.79.227.8:4330 , piece_id=YC5DPQGDIEEFVMTWYNIJSOVGR7XAOPFRHKNDZ7XM7C4L4A6GLTEQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.164387442 (600h11m41.035470487s) [noise=true , address=145.79.227.9:4366 , piece_id=U4RFVWUOI2EJF7OL2TWD4UMGLYMWNCMHY3DIP4ACZPQWT2EVIGAA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.213200708 (600h11m41.084283753s) [noise=true , address=145.79.227.20:4439 , piece_id=SVLQSJMN5BRGDVF3MEUU3DCRFEIZDYATI7J5YFXOQAC7N52KV7SA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.289985748 (600h11m41.161068803s) [noise=true , address=145.79.227.36:4518 , piece_id=7QHEQ6SF6UG4KN4G2VE5L5ZWH5W4FOKJAAGN7ZZWXPNSAG46PLHQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.335807599 (600h11m41.206890634s) [noise=true , address=77.90.31.19:4842 , piece_id=JKB7C5GUF7JXZUEZVKHS632UCPGGXR7PEUPR72KFQHK2GJOLTFVA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.340383422 (600h11m41.211466467s) [noise=true , address=77.90.31.13:4629 , piece_id=IBEWLBAIKC2D6VULA2IDCUCZ5KANCOHRLI3L2BIAYY5PYXTYYSFQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.390789361 (600h11m41.261872406s) [noise=true , address=77.90.31.11:4556 , piece_id=N5PQIQAJJQLW3LHCPM3DTZKHXN63FVDB5R2F2NPUATZUL2W3M3NQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.512772514 (600h11m41.383855559s) [noise=true , address=45.91.116.14:4189 , piece_id=OGZTOYHWHPYF7VVPNKRJIQ7C5QKO3S4SVHFXBCZKXVTDTQJUGYJA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.532982778 (600h11m41.404065823s) [noise=true , address=45.91.116.9:4263 , piece_id=2AVL5BO4LFFX5EY7B4AREQKM3CYSQ2FOL767UQXKTUZUQKSHKO5A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:41.620303829 (600h11m41.491386874s) [noise=true , address=45.91.116.7:4009 , piece_id=PYTE5AXN4WRIVUXYTK6LLOR6QRHAO2NEWJUKPSREMBCKK75W4DMQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138912742 (600h11m42.009995787s) [noise=true , address=77.90.31.10:4517 , piece_id=QW23J7OCNOFDYOWE26LXIF6ND6ZT472TFPTXIFRRRNOCMYRP2OBQ, seg_pos={0 0}, upload=526044]: Done uploading piece. err: <nil>
Dec 30 15:40:42.138937430 (600h11m42.010020475s) [seg_pos={0 0}, upload=526044]: Segment reached optimal threshold of 13 pieces.
Dec 30 15:40:42.139125157 (600h11m42.010208202s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139132431 (600h11m42.010215466s) [noise=true , address=145.79.227.49:4337 , piece_id=XIOE4RNPJBR7U3HYRDOHJPUYLI2VPLIUPSIIAERGTQZ2KPP5LHYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139155505 (600h11m42.01023854s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 15:40:42.139164863 (600h11m42.010247908s) [noise=true , address=77.90.31.18:4808 , piece_id=ZCSSIK67G7VNHNHLISED46HUNQ7LGZIEDXZ4GLLA2H3WCMU26L4A, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139311212 (600h11m42.010394267s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42292->77.90.31.17:4770: use of closed network connection
Dec 30 15:40:42.139320389 (600h11m42.010403424s) [noise=true , address=77.90.31.17:4770 , piece_id=QMPHJB3XFF5VUV5BYR72QQLO5RGEJBO2YOCWCBGDO2EGLFJ2DJYA, seg_pos={0 0}, upload=526044]: Long tail cancellation due to optimal threshold
Dec 30 15:40:42.139340818 (600h11m42.010423853s) [seg_pos={0 0}, upload=526044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 15:40:42.139420510 (600h11m42.010503555s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:40:42.139502656 (600h11m42.010585691s) [upload=526044]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:40:42.448963161 (600h11m42.320046206s) [upload=526044]: Done uploading segments...
Dec 30 15:40:42.448994711 (600h11m42.320077756s) [upload=526044]: Upload finished. err: <nil>
Duration: 22.064862614s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X..................................X.......................] duration:
6.674978856s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:43:40 UTC)
orbiter warning: context deadline exceeded
Duration: 6.67500692s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......X............X............................X..XX..X....] duration:
39.34857147s trace:
https://jaeger.dev.storj.io/trace/3b27a789ef618091
FAIL (2025-12-30 15:43:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece...
Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment
Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments...
Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil>
Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces.
Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection
Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF
Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold
Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments...
Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil>
Duration: 39.348587632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.XXXXX.XXX.XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXX] duration:
5.001102215s trace:
https://jaeger.dev.storj.io/trace/6c90cefceb6e9896
FAIL (2025-12-30 15:43:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001125429s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X.X..................................X......................] duration:
12.986182742s trace:
https://jaeger.dev.storj.io/trace/547648a2cfce50e7
FAIL (2025-12-30 15:43:27 UTC)
orbiter warning: context deadline exceeded
Duration: 12.986209284s
edge
gateway
us-select-1
|