Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
12.048125546s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:12:42 UTC)
orbiter warning: context deadline exceeded
Duration: 12.048155624s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.....X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X....X....] duration:
38.981398706s trace:
https://jaeger.dev.storj.io/trace/63bb4630073c5866
FAIL (2025-12-26 14:12:29 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
63018s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.055767421 (502h39m1.926850466s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056210935 (502h39m1.92729398s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056658958 (502h39m1.927742003s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056675941 (502h39m1.927758986s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057134323 (502h39m1.928217369s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057285190 (502h39m1.928368235s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057787948 (502h39m1.928870993s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058251400 (502h39m1.929334446s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058454658 (502h39m1.929537703s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058728569 (502h39m1.929811614s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059135984 (502h39m1.93021903s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059215707 (502h39m1.930298752s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059640646 (502h39m1.930723691s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.061148707 (502h39m1.932231752s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:03.534188993 (502h39m3.405272038s) [upload=441114]: Next returned nil segment
Dec 26 14:08:03.534209692 (502h39m3.405292727s) [upload=441114]: Waiting for error group managing segments...
Dec 26 14:08:03.922640104 (502h39m3.793723149s) [noise=true , address=145.79.227.52:4449 , piece_id=6EG4PZM53NS746KAVCOY7NKG444SL7KJFNIIMGVAYKULID3A3UMQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:03.955431504 (502h39m3.826514549s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.164774431 (502h39m4.035857476s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.202756856 (502h39m4.073839901s) [noise=true , address=77.90.31.15:4726 , piece_id=BMG2HGGES7UO4EGXBOQRGZNXEHUZZ4FAX3QFWV3ALAE34WFIO3QQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.220208919 (502h39m4.091291964s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.226575091 (502h39m4.097658136s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.300871024 (502h39m4.171954069s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.353241546 (502h39m4.224324591s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.374379137 (502h39m4.245462182s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.435669715 (502h39m4.30675275s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.836335655 (502h39m4.70741869s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.906284380 (502h39m4.777367425s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.988447744 (502h39m4.859530779s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.370986566 (502h39m5.242069601s) [noise=true , address=77.90.31.18:4812 , piece_id=PKYL22EHLA3NUNSJ67Y3DLJNNSUK43UFCLTU35DVJXWKWJBG6EIQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.371010622 (502h39m5.242093667s) [seg_pos={0 0}, upload=441114]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:08:05.371217336 (502h39m5.242300381s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection
Dec 26 14:08:05.371224960 (502h39m5.242307995s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371275928 (502h39m5.242358973s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371287549 (502h39m5.242370594s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371352844 (502h39m5.242435889s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection
Dec 26 14:08:05.371360949 (502h39m5.242443994s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371532897 (502h39m5.242615942s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371541002 (502h39m5.242624047s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371641513 (502h39m5.242724558s) [seg_pos={0 0}, upload=441114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:08:05.371748206 (502h39m5.242831251s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:08:05.371764388 (502h39m5.242847423s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:08:05.668404995 (502h39m5.53948803s) [upload=441114]: Done uploading segments...
Dec 26 14:08:05.668424543 (502h39m5.539507588s) [upload=441114]: Upload finished. err: <nil>
Duration: 38.98142692s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
10.614635063s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:12:23 UTC)
orbiter warning: context deadline exceeded
Duration: 10.614662083s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
15.445368741s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:11:58 UTC)
orbiter warning: context deadline exceeded
Duration: 15.445399561s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00103258s trace:
https://jaeger.dev.storj.io/trace/3308bcd7ea1dba4e
FAIL (2025-12-26 14:11:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001135035s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [...X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..X.X...] duration:
5.750272847s trace:
https://jaeger.dev.storj.io/trace/38874f6bade76bcc
FAIL (2025-12-26 14:11:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q) context deadline exceeded
======LOGS======
Dec 26 14:09:05.393026130 (502h40m5.264109175s) [upload=441130]: Starting upload
Dec 26 14:09:05.393112405 (502h40m5.26419545s) [upload=441130]: Done starting upload
Dec 26 14:09:05.393186145 (502h40m5.26426919s) [upload=441130]: Uploading segments...
Dec 26 14:09:05.393481497 (502h40m5.264564542s) [upload=441130]: Got next segment. Inline: false
Dec 26 14:09:05.393530521 (502h40m5.264613566s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:09:05.393536923 (502h40m5.264619978s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:09:05.639771008 (502h40m5.510854053s) [seg_pos={0 0}, upload=441130]: Begin upload segment...
Dec 26 14:09:05.639832805 (502h40m5.51091584s) [seg_pos={0 0}, upload=441130]: Done begin upload segment.
Dec 26 14:09:05.639870637 (502h40m5.510953682s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.639919089 (502h40m5.511002135s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640509614 (502h40m5.511592659s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640946135 (502h40m5.51202918s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.641477417 (502h40m5.512560462s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642020751 (502h40m5.513103786s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642520894 (502h40m5.513603939s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.643079758 (502h40m5.514162803s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644459304 (502h40m5.515542349s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644760498 (502h40m5.515843543s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646187676 (502h40m5.517270721s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646561268 (502h40m5.517644313s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.648091191 (502h40m5.519174236s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.650844914 (502h40m5.521927949s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.651984965 (502h40m5.52306801s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.654063773 (502h40m5.525146818s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:07.363859032 (502h40m7.234942068s) [upload=441130]: Next returned nil segment
Dec 26 14:09:07.363876044 (502h40m7.234959079s) [upload=441130]: Waiting for error group managing segments...
Dec 26 14:09:07.650655415 (502h40m7.52173845s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.653944187 (502h40m7.525027233s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.654306717 (502h40m7.525389762s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.656371449 (502h40m7.527454494s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.658937365 (502h40m7.5300204s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.660512294 (502h40m7.531595339s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.670889814 (502h40m7.541972849s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.675610093 (502h40m7.546693138s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.676207019 (502h40m7.547290054s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.678604855 (502h40m7.54968789s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.187764667 (502h40m8.058847712s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.469562532 (502h40m8.340645597s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565705074 (502h40m8.436788119s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565727126 (502h40m8.436810171s) [seg_pos={0 0}, upload=441130]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:09:08.566016827 (502h40m8.437099872s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:33834->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:09:08.566023971 (502h40m8.437107006s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566247888 (502h40m8.437330933s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566254079 (502h40m8.437337124s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566359179 (502h40m8.437442214s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566364389 (502h40m8.437447424s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566379068 (502h40m8.437462113s) [seg_pos={0 0}, upload=441130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:09:08.566450574 (502h40m8.437533619s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:09:08.566454892 (502h40m8.437537947s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:09:08.863827874 (502h40m8.734910919s) [upload=441130]: Done uploading segments...
Dec 26 14:09:08.863859003 (502h40m8.734942038s) [upload=441130]: Upload finished. err: <nil>
Duration: 5.750297605s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
11.903693753s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:11:50 UTC)
orbiter warning: context deadline exceeded
Duration: 11.903718189s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [......X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X....X...] duration:
34.749366345s trace:
https://jaeger.dev.storj.io/trace/63bb4630073c5866
FAIL (2025-12-26 14:11:26 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
63018s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.055767421 (502h39m1.926850466s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056210935 (502h39m1.92729398s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056658958 (502h39m1.927742003s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056675941 (502h39m1.927758986s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057134323 (502h39m1.928217369s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057285190 (502h39m1.928368235s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057787948 (502h39m1.928870993s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058251400 (502h39m1.929334446s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058454658 (502h39m1.929537703s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058728569 (502h39m1.929811614s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059135984 (502h39m1.93021903s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059215707 (502h39m1.930298752s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059640646 (502h39m1.930723691s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.061148707 (502h39m1.932231752s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:03.534188993 (502h39m3.405272038s) [upload=441114]: Next returned nil segment
Dec 26 14:08:03.534209692 (502h39m3.405292727s) [upload=441114]: Waiting for error group managing segments...
Dec 26 14:08:03.922640104 (502h39m3.793723149s) [noise=true , address=145.79.227.52:4449 , piece_id=6EG4PZM53NS746KAVCOY7NKG444SL7KJFNIIMGVAYKULID3A3UMQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:03.955431504 (502h39m3.826514549s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.164774431 (502h39m4.035857476s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.202756856 (502h39m4.073839901s) [noise=true , address=77.90.31.15:4726 , piece_id=BMG2HGGES7UO4EGXBOQRGZNXEHUZZ4FAX3QFWV3ALAE34WFIO3QQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.220208919 (502h39m4.091291964s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.226575091 (502h39m4.097658136s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.300871024 (502h39m4.171954069s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.353241546 (502h39m4.224324591s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.374379137 (502h39m4.245462182s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.435669715 (502h39m4.30675275s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.836335655 (502h39m4.70741869s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.906284380 (502h39m4.777367425s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.988447744 (502h39m4.859530779s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.370986566 (502h39m5.242069601s) [noise=true , address=77.90.31.18:4812 , piece_id=PKYL22EHLA3NUNSJ67Y3DLJNNSUK43UFCLTU35DVJXWKWJBG6EIQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.371010622 (502h39m5.242093667s) [seg_pos={0 0}, upload=441114]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:08:05.371217336 (502h39m5.242300381s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection
Dec 26 14:08:05.371224960 (502h39m5.242307995s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371275928 (502h39m5.242358973s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371287549 (502h39m5.242370594s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371352844 (502h39m5.242435889s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection
Dec 26 14:08:05.371360949 (502h39m5.242443994s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371532897 (502h39m5.242615942s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371541002 (502h39m5.242624047s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371641513 (502h39m5.242724558s) [seg_pos={0 0}, upload=441114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:08:05.371748206 (502h39m5.242831251s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:08:05.371764388 (502h39m5.242847423s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:08:05.668404995 (502h39m5.53948803s) [upload=441114]: Done uploading segments...
Dec 26 14:08:05.668424543 (502h39m5.539507588s) [upload=441114]: Upload finished. err: <nil>
Duration: 34.749383729s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [....X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..X.X..] duration:
5.904599542s trace:
https://jaeger.dev.storj.io/trace/38874f6bade76bcc
FAIL (2025-12-26 14:11:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q) context deadline exceeded
======LOGS======
Dec 26 14:09:05.393026130 (502h40m5.264109175s) [upload=441130]: Starting upload
Dec 26 14:09:05.393112405 (502h40m5.26419545s) [upload=441130]: Done starting upload
Dec 26 14:09:05.393186145 (502h40m5.26426919s) [upload=441130]: Uploading segments...
Dec 26 14:09:05.393481497 (502h40m5.264564542s) [upload=441130]: Got next segment. Inline: false
Dec 26 14:09:05.393530521 (502h40m5.264613566s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:09:05.393536923 (502h40m5.264619978s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:09:05.639771008 (502h40m5.510854053s) [seg_pos={0 0}, upload=441130]: Begin upload segment...
Dec 26 14:09:05.639832805 (502h40m5.51091584s) [seg_pos={0 0}, upload=441130]: Done begin upload segment.
Dec 26 14:09:05.639870637 (502h40m5.510953682s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.639919089 (502h40m5.511002135s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640509614 (502h40m5.511592659s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640946135 (502h40m5.51202918s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.641477417 (502h40m5.512560462s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642020751 (502h40m5.513103786s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642520894 (502h40m5.513603939s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.643079758 (502h40m5.514162803s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644459304 (502h40m5.515542349s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644760498 (502h40m5.515843543s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646187676 (502h40m5.517270721s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646561268 (502h40m5.517644313s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.648091191 (502h40m5.519174236s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.650844914 (502h40m5.521927949s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.651984965 (502h40m5.52306801s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.654063773 (502h40m5.525146818s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:07.363859032 (502h40m7.234942068s) [upload=441130]: Next returned nil segment
Dec 26 14:09:07.363876044 (502h40m7.234959079s) [upload=441130]: Waiting for error group managing segments...
Dec 26 14:09:07.650655415 (502h40m7.52173845s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.653944187 (502h40m7.525027233s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.654306717 (502h40m7.525389762s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.656371449 (502h40m7.527454494s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.658937365 (502h40m7.5300204s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.660512294 (502h40m7.531595339s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.670889814 (502h40m7.541972849s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.675610093 (502h40m7.546693138s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.676207019 (502h40m7.547290054s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.678604855 (502h40m7.54968789s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.187764667 (502h40m8.058847712s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.469562532 (502h40m8.340645597s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565705074 (502h40m8.436788119s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565727126 (502h40m8.436810171s) [seg_pos={0 0}, upload=441130]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:09:08.566016827 (502h40m8.437099872s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:33834->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:09:08.566023971 (502h40m8.437107006s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566247888 (502h40m8.437330933s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566254079 (502h40m8.437337124s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566359179 (502h40m8.437442214s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566364389 (502h40m8.437447424s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566379068 (502h40m8.437462113s) [seg_pos={0 0}, upload=441130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:09:08.566450574 (502h40m8.437533619s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:09:08.566454892 (502h40m8.437537947s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:09:08.863827874 (502h40m8.734910919s) [upload=441130]: Done uploading segments...
Dec 26 14:09:08.863859003 (502h40m8.734942038s) [upload=441130]: Upload finished. err: <nil>
Duration: 5.904634339s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
12.591312637s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:11:06 UTC)
orbiter warning: context deadline exceeded
Duration: 12.591331451s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001429278s trace:
https://jaeger.dev.storj.io/trace/2023eba41edcf5d1
FAIL (2025-12-26 14:10:55 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00145181s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.......X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X....X..] duration:
55.299271715s trace:
https://jaeger.dev.storj.io/trace/63bb4630073c5866
FAIL (2025-12-26 14:10:51 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
63018s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.055767421 (502h39m1.926850466s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056210935 (502h39m1.92729398s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056658958 (502h39m1.927742003s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056675941 (502h39m1.927758986s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057134323 (502h39m1.928217369s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057285190 (502h39m1.928368235s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057787948 (502h39m1.928870993s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058251400 (502h39m1.929334446s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058454658 (502h39m1.929537703s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058728569 (502h39m1.929811614s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059135984 (502h39m1.93021903s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059215707 (502h39m1.930298752s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059640646 (502h39m1.930723691s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.061148707 (502h39m1.932231752s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:03.534188993 (502h39m3.405272038s) [upload=441114]: Next returned nil segment
Dec 26 14:08:03.534209692 (502h39m3.405292727s) [upload=441114]: Waiting for error group managing segments...
Dec 26 14:08:03.922640104 (502h39m3.793723149s) [noise=true , address=145.79.227.52:4449 , piece_id=6EG4PZM53NS746KAVCOY7NKG444SL7KJFNIIMGVAYKULID3A3UMQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:03.955431504 (502h39m3.826514549s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.164774431 (502h39m4.035857476s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.202756856 (502h39m4.073839901s) [noise=true , address=77.90.31.15:4726 , piece_id=BMG2HGGES7UO4EGXBOQRGZNXEHUZZ4FAX3QFWV3ALAE34WFIO3QQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.220208919 (502h39m4.091291964s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.226575091 (502h39m4.097658136s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.300871024 (502h39m4.171954069s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.353241546 (502h39m4.224324591s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.374379137 (502h39m4.245462182s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.435669715 (502h39m4.30675275s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.836335655 (502h39m4.70741869s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.906284380 (502h39m4.777367425s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.988447744 (502h39m4.859530779s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.370986566 (502h39m5.242069601s) [noise=true , address=77.90.31.18:4812 , piece_id=PKYL22EHLA3NUNSJ67Y3DLJNNSUK43UFCLTU35DVJXWKWJBG6EIQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.371010622 (502h39m5.242093667s) [seg_pos={0 0}, upload=441114]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:08:05.371217336 (502h39m5.242300381s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection
Dec 26 14:08:05.371224960 (502h39m5.242307995s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371275928 (502h39m5.242358973s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371287549 (502h39m5.242370594s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371352844 (502h39m5.242435889s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection
Dec 26 14:08:05.371360949 (502h39m5.242443994s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371532897 (502h39m5.242615942s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371541002 (502h39m5.242624047s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371641513 (502h39m5.242724558s) [seg_pos={0 0}, upload=441114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:08:05.371748206 (502h39m5.242831251s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:08:05.371764388 (502h39m5.242847423s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:08:05.668404995 (502h39m5.53948803s) [upload=441114]: Done uploading segments...
Dec 26 14:08:05.668424543 (502h39m5.539507588s) [upload=441114]: Upload finished. err: <nil>
Duration: 55.299287576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
14.022299222s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:10:47 UTC)
orbiter warning: context deadline exceeded
Duration: 14.022319369s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.....X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..X.X.] duration:
5.857974339s trace:
https://jaeger.dev.storj.io/trace/38874f6bade76bcc
FAIL (2025-12-26 14:10:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q) context deadline exceeded
======LOGS======
Dec 26 14:09:05.393026130 (502h40m5.264109175s) [upload=441130]: Starting upload
Dec 26 14:09:05.393112405 (502h40m5.26419545s) [upload=441130]: Done starting upload
Dec 26 14:09:05.393186145 (502h40m5.26426919s) [upload=441130]: Uploading segments...
Dec 26 14:09:05.393481497 (502h40m5.264564542s) [upload=441130]: Got next segment. Inline: false
Dec 26 14:09:05.393530521 (502h40m5.264613566s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:09:05.393536923 (502h40m5.264619978s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:09:05.639771008 (502h40m5.510854053s) [seg_pos={0 0}, upload=441130]: Begin upload segment...
Dec 26 14:09:05.639832805 (502h40m5.51091584s) [seg_pos={0 0}, upload=441130]: Done begin upload segment.
Dec 26 14:09:05.639870637 (502h40m5.510953682s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.639919089 (502h40m5.511002135s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640509614 (502h40m5.511592659s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640946135 (502h40m5.51202918s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.641477417 (502h40m5.512560462s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642020751 (502h40m5.513103786s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642520894 (502h40m5.513603939s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.643079758 (502h40m5.514162803s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644459304 (502h40m5.515542349s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644760498 (502h40m5.515843543s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646187676 (502h40m5.517270721s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646561268 (502h40m5.517644313s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.648091191 (502h40m5.519174236s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.650844914 (502h40m5.521927949s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.651984965 (502h40m5.52306801s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.654063773 (502h40m5.525146818s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:07.363859032 (502h40m7.234942068s) [upload=441130]: Next returned nil segment
Dec 26 14:09:07.363876044 (502h40m7.234959079s) [upload=441130]: Waiting for error group managing segments...
Dec 26 14:09:07.650655415 (502h40m7.52173845s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.653944187 (502h40m7.525027233s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.654306717 (502h40m7.525389762s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.656371449 (502h40m7.527454494s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.658937365 (502h40m7.5300204s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.660512294 (502h40m7.531595339s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.670889814 (502h40m7.541972849s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.675610093 (502h40m7.546693138s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.676207019 (502h40m7.547290054s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.678604855 (502h40m7.54968789s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.187764667 (502h40m8.058847712s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.469562532 (502h40m8.340645597s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565705074 (502h40m8.436788119s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565727126 (502h40m8.436810171s) [seg_pos={0 0}, upload=441130]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:09:08.566016827 (502h40m8.437099872s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:33834->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:09:08.566023971 (502h40m8.437107006s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566247888 (502h40m8.437330933s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566254079 (502h40m8.437337124s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566359179 (502h40m8.437442214s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566364389 (502h40m8.437447424s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566379068 (502h40m8.437462113s) [seg_pos={0 0}, upload=441130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:09:08.566450574 (502h40m8.437533619s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:09:08.566454892 (502h40m8.437537947s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:09:08.863827874 (502h40m8.734910919s) [upload=441130]: Done uploading segments...
Dec 26 14:09:08.863859003 (502h40m8.734942038s) [upload=441130]: Upload finished. err: <nil>
Duration: 5.858017082s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001223781s trace:
https://jaeger.dev.storj.io/trace/30f27e87cae01ba
FAIL (2025-12-26 14:10:18 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001251364s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
7.043254177s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:10:12 UTC)
orbiter warning: context deadline exceeded
Duration: 7.043303761s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
27.456793279s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:10:07 UTC)
orbiter warning: context deadline exceeded
Duration: 27.45681947s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [........X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X....X.] duration:
55.019787541s trace:
https://jaeger.dev.storj.io/trace/63bb4630073c5866
FAIL (2025-12-26 14:09:56 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
63018s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.055767421 (502h39m1.926850466s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056210935 (502h39m1.92729398s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056658958 (502h39m1.927742003s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056675941 (502h39m1.927758986s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057134323 (502h39m1.928217369s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057285190 (502h39m1.928368235s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057787948 (502h39m1.928870993s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058251400 (502h39m1.929334446s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058454658 (502h39m1.929537703s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058728569 (502h39m1.929811614s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059135984 (502h39m1.93021903s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059215707 (502h39m1.930298752s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059640646 (502h39m1.930723691s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.061148707 (502h39m1.932231752s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:03.534188993 (502h39m3.405272038s) [upload=441114]: Next returned nil segment
Dec 26 14:08:03.534209692 (502h39m3.405292727s) [upload=441114]: Waiting for error group managing segments...
Dec 26 14:08:03.922640104 (502h39m3.793723149s) [noise=true , address=145.79.227.52:4449 , piece_id=6EG4PZM53NS746KAVCOY7NKG444SL7KJFNIIMGVAYKULID3A3UMQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:03.955431504 (502h39m3.826514549s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.164774431 (502h39m4.035857476s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.202756856 (502h39m4.073839901s) [noise=true , address=77.90.31.15:4726 , piece_id=BMG2HGGES7UO4EGXBOQRGZNXEHUZZ4FAX3QFWV3ALAE34WFIO3QQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.220208919 (502h39m4.091291964s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.226575091 (502h39m4.097658136s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.300871024 (502h39m4.171954069s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.353241546 (502h39m4.224324591s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.374379137 (502h39m4.245462182s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.435669715 (502h39m4.30675275s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.836335655 (502h39m4.70741869s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.906284380 (502h39m4.777367425s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.988447744 (502h39m4.859530779s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.370986566 (502h39m5.242069601s) [noise=true , address=77.90.31.18:4812 , piece_id=PKYL22EHLA3NUNSJ67Y3DLJNNSUK43UFCLTU35DVJXWKWJBG6EIQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.371010622 (502h39m5.242093667s) [seg_pos={0 0}, upload=441114]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:08:05.371217336 (502h39m5.242300381s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection
Dec 26 14:08:05.371224960 (502h39m5.242307995s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371275928 (502h39m5.242358973s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371287549 (502h39m5.242370594s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371352844 (502h39m5.242435889s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection
Dec 26 14:08:05.371360949 (502h39m5.242443994s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371532897 (502h39m5.242615942s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371541002 (502h39m5.242624047s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371641513 (502h39m5.242724558s) [seg_pos={0 0}, upload=441114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:08:05.371748206 (502h39m5.242831251s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:08:05.371764388 (502h39m5.242847423s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:08:05.668404995 (502h39m5.53948803s) [upload=441114]: Done uploading segments...
Dec 26 14:08:05.668424543 (502h39m5.539507588s) [upload=441114]: Upload finished. err: <nil>
Duration: 55.019804423s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [......X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..X.X] duration:
30.004481924s trace:
https://jaeger.dev.storj.io/trace/38874f6bade76bcc
FAIL (2025-12-26 14:09:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q) context deadline exceeded
======LOGS======
Dec 26 14:09:05.393026130 (502h40m5.264109175s) [upload=441130]: Starting upload
Dec 26 14:09:05.393112405 (502h40m5.26419545s) [upload=441130]: Done starting upload
Dec 26 14:09:05.393186145 (502h40m5.26426919s) [upload=441130]: Uploading segments...
Dec 26 14:09:05.393481497 (502h40m5.264564542s) [upload=441130]: Got next segment. Inline: false
Dec 26 14:09:05.393530521 (502h40m5.264613566s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:09:05.393536923 (502h40m5.264619978s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:09:05.639771008 (502h40m5.510854053s) [seg_pos={0 0}, upload=441130]: Begin upload segment...
Dec 26 14:09:05.639832805 (502h40m5.51091584s) [seg_pos={0 0}, upload=441130]: Done begin upload segment.
Dec 26 14:09:05.639870637 (502h40m5.510953682s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.639919089 (502h40m5.511002135s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640509614 (502h40m5.511592659s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.640946135 (502h40m5.51202918s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.641477417 (502h40m5.512560462s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642020751 (502h40m5.513103786s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.642520894 (502h40m5.513603939s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.643079758 (502h40m5.514162803s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644459304 (502h40m5.515542349s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.644760498 (502h40m5.515843543s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646187676 (502h40m5.517270721s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.646561268 (502h40m5.517644313s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.648091191 (502h40m5.519174236s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.650844914 (502h40m5.521927949s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.651984965 (502h40m5.52306801s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:05.654063773 (502h40m5.525146818s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Uploading piece...
Dec 26 14:09:07.363859032 (502h40m7.234942068s) [upload=441130]: Next returned nil segment
Dec 26 14:09:07.363876044 (502h40m7.234959079s) [upload=441130]: Waiting for error group managing segments...
Dec 26 14:09:07.650655415 (502h40m7.52173845s) [noise=true , address=145.79.227.17:4971 , piece_id=UIWFYVF7ZYVNXURZKYY6PCAAFHJHC77JO4RHSHACBZKWWRDNRA4A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.653944187 (502h40m7.525027233s) [noise=true , address=145.79.227.43:4158 , piece_id=CXSVRUTMHPR7F4PUWCHDSVQJMZMFGWE62I3QK62EQGRBP3VZ2P7A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.654306717 (502h40m7.525389762s) [noise=true , address=145.79.227.5:4222 , piece_id=UYKAXFEOW3ELJMYJY6XCIL4735Y3KROJWDYYXW4UVIZMLSXHOOEQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.656371449 (502h40m7.527454494s) [noise=true , address=145.79.227.42:4121 , piece_id=3D6UHSDFJN7C5TXVQ4YSINV47VJSOWG5CQDYWMV6YI4C4RVBL7MA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.658937365 (502h40m7.5300204s) [noise=true , address=145.79.227.23:4546 , piece_id=IWXQ25GWWWTRBNX5XSC2GBBYGIFXVV2PIZHCDUR5UQVMEMNF5WEA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.660512294 (502h40m7.531595339s) [noise=true , address=145.79.227.13:4803 , piece_id=TSTOWNRECZAX3Z4URJRM7CRSVPYO4XG3VFKRQ7MFO6VN64S7JSDQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.670889814 (502h40m7.541972849s) [noise=true , address=145.79.227.46:4602 , piece_id=TMNDUY7JQ3EJL5ENHHWBZD5QKUNDODDJHQRHOYPHQDWOJ4FHVGBA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.675610093 (502h40m7.546693138s) [noise=true , address=145.79.227.29:4943 , piece_id=IU6EMW5VP6PORPVFZKNXM3FDNEOYO3LQSW7WIFO2T4ZPH623JX5A, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.676207019 (502h40m7.547290054s) [noise=true , address=145.79.227.11:4730 , piece_id=TEUAY2POYI5D5ADOTHT63ESJB6ITTKF4X4K3N27MD6VULF35A62Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:07.678604855 (502h40m7.54968789s) [noise=true , address=145.79.227.37:4565 , piece_id=IFEA4I2UT5CMMISGHWOFKCDX2SFQ3KNNO3IXHWDUG2IMATWPH2BQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.187764667 (502h40m8.058847712s) [noise=true , address=77.90.31.18:4808 , piece_id=X5HZR7AA35K2777XOAUMUGOCS77J77IXEZT7JHCRQO376OYUX3WQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.469562532 (502h40m8.340645597s) [noise=true , address=45.91.116.5:4299 , piece_id=HZGUV5IJGFWWMTUO5LNE66QAY3XATEGM2WH3H3LV2S4UZHSSSYNA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565705074 (502h40m8.436788119s) [noise=true , address=45.91.116.6:4974 , piece_id=MM45NDBM26TWET3NM7CFAESXJD7HFA4QGEEUZYZUMEKDBALGDX6Q, seg_pos={0 0}, upload=441130]: Done uploading piece. err: <nil>
Dec 26 14:09:08.565727126 (502h40m8.436810171s) [seg_pos={0 0}, upload=441130]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:09:08.566016827 (502h40m8.437099872s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:33834->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:09:08.566023971 (502h40m8.437107006s) [noise=true , address=45.91.116.12:4226 , piece_id=VALUTUVP4AKUWWHDEHFI2HTYBIRABTKSFJJVEPEN4VBGYAG66NYQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566247888 (502h40m8.437330933s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566254079 (502h40m8.437337124s) [noise=true , address=77.90.31.10:4517 , piece_id=FU5ETS44N5TI6766M5RL6SSOBDTYAQR6N5MXWGE4RF3KBVLSHSBQ, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566359179 (502h40m8.437442214s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 26 14:09:08.566364389 (502h40m8.437447424s) [noise=true , address=45.91.116.11:4117 , piece_id=3HRQMKMSFDOTW2L5JM2RYY6AU6WCZQTYCPO2PLJJH5FF4UBHHXYA, seg_pos={0 0}, upload=441130]: Long tail cancellation due to optimal threshold
Dec 26 14:09:08.566379068 (502h40m8.437462113s) [seg_pos={0 0}, upload=441130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:09:08.566450574 (502h40m8.437533619s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:09:08.566454892 (502h40m8.437537947s) [upload=441130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:09:08.863827874 (502h40m8.734910919s) [upload=441130]: Done uploading segments...
Dec 26 14:09:08.863859003 (502h40m8.734942038s) [upload=441130]: Upload finished. err: <nil>
Duration: 30.004507812s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [.X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000949581s trace:
https://jaeger.dev.storj.io/trace/803a686abad599e
FAIL (2025-12-26 14:09:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000964179s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
13.275577346s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:09:13 UTC)
orbiter warning: context deadline exceeded
Duration: 13.275601362s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [X........X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X....X] duration:
1m0.003390353s trace:
https://jaeger.dev.storj.io/trace/63bb4630073c5866
FAIL (2025-12-26 14:09:01 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
63018s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.055767421 (502h39m1.926850466s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056210935 (502h39m1.92729398s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056658958 (502h39m1.927742003s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.056675941 (502h39m1.927758986s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057134323 (502h39m1.928217369s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057285190 (502h39m1.928368235s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.057787948 (502h39m1.928870993s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058251400 (502h39m1.929334446s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058454658 (502h39m1.929537703s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.058728569 (502h39m1.929811614s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059135984 (502h39m1.93021903s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059215707 (502h39m1.930298752s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.059640646 (502h39m1.930723691s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:02.061148707 (502h39m1.932231752s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Uploading piece...
Dec 26 14:08:03.534188993 (502h39m3.405272038s) [upload=441114]: Next returned nil segment
Dec 26 14:08:03.534209692 (502h39m3.405292727s) [upload=441114]: Waiting for error group managing segments...
Dec 26 14:08:03.922640104 (502h39m3.793723149s) [noise=true , address=145.79.227.52:4449 , piece_id=6EG4PZM53NS746KAVCOY7NKG444SL7KJFNIIMGVAYKULID3A3UMQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:03.955431504 (502h39m3.826514549s) [noise=true , address=145.79.227.34:4909 , piece_id=NX76M6GLHPRYVJKH6LIAWXO6QTLLPALHBCK2JE2736K3N7ASKN2Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.164774431 (502h39m4.035857476s) [noise=true , address=45.91.116.5:4327 , piece_id=62437VJ5D57ODJN3KXKCN4R7LHLW72I4QZEK6UD2V4B63RRCQXSA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.202756856 (502h39m4.073839901s) [noise=true , address=77.90.31.15:4726 , piece_id=BMG2HGGES7UO4EGXBOQRGZNXEHUZZ4FAX3QFWV3ALAE34WFIO3QQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.220208919 (502h39m4.091291964s) [noise=true , address=77.90.31.13:4634 , piece_id=OIJ2T7AULTDNI6LLJL4KMEY4EGSKVBAQXQ2JQL6YHBV7C4Q3HSCQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.226575091 (502h39m4.097658136s) [noise=true , address=77.90.31.11:4558 , piece_id=JVS36D5Y2IN6IV74PRYZ5Y5R4Q2N563W7HFWAXXPZPAD5DV2XO4A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.300871024 (502h39m4.171954069s) [noise=true , address=77.90.31.21:4905 , piece_id=GUJWZZRESQI5OE34BBO4QLEMNJK3UK4IWXDTUDUBMFT5YWASEX3Q, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.353241546 (502h39m4.224324591s) [noise=true , address=77.90.31.10:4542 , piece_id=77ZUSPHYMYP5GLQJNPCGDLI4ZD4D4MNI2UUS2BQ67MZPZLARZQYQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.374379137 (502h39m4.245462182s) [noise=true , address=77.90.31.19:4856 , piece_id=PZSMUPUPYYKNXO6R3EJQR4QZ346VFEB2O4EGRC3LBARIO4AW5TXA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.435669715 (502h39m4.30675275s) [noise=true , address=145.79.227.23:4567 , piece_id=FDK7BCTJXODLU7WYGQBW7MAJLD5QCEOJHRCORPPD7M5PKA2ZNREA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.836335655 (502h39m4.70741869s) [noise=true , address=145.79.227.30:4764 , piece_id=G3WHP7RD5CDLA6GKZLXBG7VUZ4TS3CYQSLD6RRYBF72OMZJORABQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.906284380 (502h39m4.777367425s) [noise=true , address=145.79.227.44:4218 , piece_id=MDGQJV7J2OYGMDKO4K2U74WAGQ3NOYBSX7EZAER4MCREAEJ5XKXQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:04.988447744 (502h39m4.859530779s) [noise=true , address=145.79.227.40:4070 , piece_id=7DQJLNBOCLHFGK3Q36K3ILA7LF2QLXPOPXKWEFKLJTZSLPOP3GFQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.370986566 (502h39m5.242069601s) [noise=true , address=77.90.31.18:4812 , piece_id=PKYL22EHLA3NUNSJ67Y3DLJNNSUK43UFCLTU35DVJXWKWJBG6EIQ, seg_pos={0 0}, upload=441114]: Done uploading piece. err: <nil>
Dec 26 14:08:05.371010622 (502h39m5.242093667s) [seg_pos={0 0}, upload=441114]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:08:05.371217336 (502h39m5.242300381s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47608->64.49.13.169:4707: use of closed network connection
Dec 26 14:08:05.371224960 (502h39m5.242307995s) [noise=true , address=64.49.13.169:4707 , piece_id=67CNBTULM2WTZ24AL7RWGCJCLLW2IOASOI4IBJNXWR2CNTKWX2JA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371275928 (502h39m5.242358973s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kSxoZb9nmG6oGNytpFTsoiGh3ZPmebVVe8TZxGzANaw7EfdeF): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371287549 (502h39m5.242370594s) [noise=true , address=77.90.31.12:4593 , piece_id=DW7EDVM3AEAOZBRFYRJYTCJCZCR2SKBUL3WH3HRXXNNSGXI5ICWA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371352844 (502h39m5.242435889s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44394->64.49.13.169:4706: use of closed network connection
Dec 26 14:08:05.371360949 (502h39m5.242443994s) [noise=true , address=64.49.13.169:4706 , piece_id=TA5CP4CNC547CICUTNX4IDPYYUMBPGHYKVVIAE5BYA3XITO4RTDA, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371532897 (502h39m5.242615942s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJVCqcpsnRfG6zJwpd4Bkm4dKX7DV5AFEesUcRs38VUPuxS3z): protocol: expected piece hash; context canceled; EOF
Dec 26 14:08:05.371541002 (502h39m5.242624047s) [noise=true , address=64.49.13.155:4923 , piece_id=XPHHYCZ6GCPBXB4IWLKWRHKLWUX5NYFAIL54OOTIZIWIPMOQQH6A, seg_pos={0 0}, upload=441114]: Long tail cancellation due to optimal threshold
Dec 26 14:08:05.371641513 (502h39m5.242724558s) [seg_pos={0 0}, upload=441114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:08:05.371748206 (502h39m5.242831251s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:08:05.371764388 (502h39m5.242847423s) [upload=441114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:08:05.668404995 (502h39m5.53948803s) [upload=441114]: Done uploading segments...
Dec 26 14:08:05.668424543 (502h39m5.539507588s) [upload=441114]: Upload finished. err: <nil>
Duration: 1m0.003403107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
25.148410129s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:09:00 UTC)
orbiter warning: context deadline exceeded
Duration: 25.148440114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXXX] duration:
5.001086313s trace:
https://jaeger.dev.storj.io/trace/68442f0718a197
FAIL (2025-12-26 14:08:52 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001109707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
13.661530291s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:08:26 UTC)
orbiter warning: context deadline exceeded
Duration: 13.661559427s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X......X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..X.] duration:
6.043758848s trace:
https://jaeger.dev.storj.io/trace/789322c1f1ca0733
FAIL (2025-12-26 14:08:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: B663A7HFTICDQSVAY6VUOEPOMIPX6Z4RTGODIULBHH7OGLOUHETQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WHUZ35F63HPQLMT4OHITVNGVRVE35LXZZYA4LICROWXWRRFBL7QQ) context deadline exceeded
======LOGS======
Dec 26 14:07:43.883075431 (502h38m43.754158476s) [upload=441108]: Starting upload
Dec 26 14:07:43.883145184 (502h38m43.754228229s) [upload=441108]: Done starting upload
Dec 26 14:07:43.883364942 (502h38m43.754447987s) [upload=441108]: Uploading segments...
Dec 26 14:07:43.883408225 (502h38m43.75449127s) [upload=441108]: Got next segment. Inline: false
Dec 26 14:07:43.883437450 (502h38m43.754520495s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:07:43.883442350 (502h38m43.754525385s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:07:44.135341702 (502h38m44.006424737s) [seg_pos={0 0}, upload=441108]: Begin upload segment...
Dec 26 14:07:44.135396447 (502h38m44.006479492s) [seg_pos={0 0}, upload=441108]: Done begin upload segment.
Dec 26 14:07:44.135421695 (502h38m44.00650473s) [noise=true , address=45.91.116.13:4156 , piece_id=6WSAT4CBID2ECUFAQBNS3H3WCFGFXIAPDOSCVPW5NSNVHOAZEM6Q, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.135437244 (502h38m44.006520289s) [noise=true , address=145.79.227.48:4303 , piece_id=CXXG5UV74SQN4XOP3AURZSEHRSQIZB3YUXPCVHW2MIEX73IHZH3Q, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.135555980 (502h38m44.006639015s) [noise=true , address=45.91.116.5:4297 , piece_id=LYVNAHVXCRNY55EFCPTX6FX2LZV4FTU67COGDVNJH5Q5MJXYD6ZQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.135947646 (502h38m44.007030691s) [noise=true , address=145.79.227.12:4767 , piece_id=EQMJCI5TJDN3TTNW2CYB44SP23AYCWHXRFEN4RBSLGHB725OXB7A, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.136332138 (502h38m44.007415173s) [noise=true , address=145.79.227.19:4077 , piece_id=HIQMV3FDMIHD64YG46FYIDVQ5LE55KD3JLZ6NY2CBL5AP4DXZLVA, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.136741207 (502h38m44.007824252s) [noise=true , address=145.79.227.39:4015 , piece_id=HPHLYNRDJG5ICQP7KYZ7HXBCRUICSTWXMMXMLPCJZ34EHD3VJDFQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.137144695 (502h38m44.00822773s) [noise=true , address=145.79.227.16:4910 , piece_id=NTVBZNP6HRJUQ5IT3IYSQM6J53OLRUVNZE2YF3XLJV2DYQV7WOMA, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.137598339 (502h38m44.008681374s) [noise=true , address=45.91.116.9:4263 , piece_id=6HPT7BHKUWTJV637IC2WWLO5P2QAJUY55MJHIIF2GFPROBBB7JBA, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.138062132 (502h38m44.009145167s) [noise=true , address=77.90.31.10:4519 , piece_id=B663A7HFTICDQSVAY6VUOEPOMIPX6Z4RTGODIULBHH7OGLOUHETQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.138467284 (502h38m44.009550319s) [noise=true , address=145.79.227.14:4840 , piece_id=7D756DMUJM7JE7VVCPMQOU4FY2XXUHNJ7DU5Y6BI5YCXNHGZJFLQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.138882655 (502h38m44.00996569s) [noise=true , address=145.79.227.21:4473 , piece_id=MBRB7BG7EOPY6EJKZZCGQMOJHM72KFRUZD6IQRTN2SP2DKVQV6KQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.139241017 (502h38m44.010324052s) [noise=true , address=145.79.227.15:4875 , piece_id=UHCEXIJU3SQBEJIX24FBZKFVBAIJ62EUPHCF3YFZTAAXZMJ5RDGQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.139306121 (502h38m44.010389156s) [noise=true , address=77.90.31.19:4844 , piece_id=WHUZ35F63HPQLMT4OHITVNGVRVE35LXZZYA4LICROWXWRRFBL7QQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.139718967 (502h38m44.010802002s) [noise=true , address=145.79.227.37:4567 , piece_id=AR4SGG3KYVJ7EONKESDS5HEQ5MT7KPTHWDNZ3MWML5OPJHC6V57Q, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.140132515 (502h38m44.01121556s) [noise=true , address=77.90.31.11:4555 , piece_id=OKWIBDSCGJ57NSWTOAJ3SRZ7MIJQTSXPRGGQNWAIMPKRNN2UEINQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.140579877 (502h38m44.011662912s) [noise=true , address=145.79.227.50:4374 , piece_id=VQQ2VH6X46ZCXHWGHFS2XIQBSPF6OWS74PEL7FZBFMTEYS52UL5A, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:45.843911697 (502h38m45.714994732s) [upload=441108]: Next returned nil segment
Dec 26 14:07:45.843932196 (502h38m45.715015241s) [upload=441108]: Waiting for error group managing segments...
Dec 26 14:07:46.127244323 (502h38m45.998327368s) [noise=true , address=145.79.227.15:4875 , piece_id=UHCEXIJU3SQBEJIX24FBZKFVBAIJ62EUPHCF3YFZTAAXZMJ5RDGQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.132567029 (502h38m46.003650084s) [noise=true , address=145.79.227.48:4303 , piece_id=CXXG5UV74SQN4XOP3AURZSEHRSQIZB3YUXPCVHW2MIEX73IHZH3Q, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.138759751 (502h38m46.009842786s) [noise=true , address=145.79.227.37:4567 , piece_id=AR4SGG3KYVJ7EONKESDS5HEQ5MT7KPTHWDNZ3MWML5OPJHC6V57Q, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.141097462 (502h38m46.012180497s) [noise=true , address=145.79.227.14:4840 , piece_id=7D756DMUJM7JE7VVCPMQOU4FY2XXUHNJ7DU5Y6BI5YCXNHGZJFLQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.142037854 (502h38m46.013120889s) [noise=true , address=145.79.227.16:4910 , piece_id=NTVBZNP6HRJUQ5IT3IYSQM6J53OLRUVNZE2YF3XLJV2DYQV7WOMA, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.143284998 (502h38m46.014368033s) [noise=true , address=145.79.227.21:4473 , piece_id=MBRB7BG7EOPY6EJKZZCGQMOJHM72KFRUZD6IQRTN2SP2DKVQV6KQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.147003078 (502h38m46.018086113s) [noise=true , address=145.79.227.12:4767 , piece_id=EQMJCI5TJDN3TTNW2CYB44SP23AYCWHXRFEN4RBSLGHB725OXB7A, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.149266107 (502h38m46.020349152s) [noise=true , address=145.79.227.19:4077 , piece_id=HIQMV3FDMIHD64YG46FYIDVQ5LE55KD3JLZ6NY2CBL5AP4DXZLVA, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.163873552 (502h38m46.034956598s) [noise=true , address=145.79.227.50:4374 , piece_id=VQQ2VH6X46ZCXHWGHFS2XIQBSPF6OWS74PEL7FZBFMTEYS52UL5A, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.175094869 (502h38m46.046177905s) [noise=true , address=145.79.227.39:4015 , piece_id=HPHLYNRDJG5ICQP7KYZ7HXBCRUICSTWXMMXMLPCJZ34EHD3VJDFQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.344919597 (502h38m46.216002642s) [noise=true , address=77.90.31.10:4519 , piece_id=B663A7HFTICDQSVAY6VUOEPOMIPX6Z4RTGODIULBHH7OGLOUHETQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.402876376 (502h38m46.273959411s) [noise=true , address=45.91.116.5:4297 , piece_id=LYVNAHVXCRNY55EFCPTX6FX2LZV4FTU67COGDVNJH5Q5MJXYD6ZQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.528360601 (502h38m46.399443636s) [noise=true , address=77.90.31.19:4844 , piece_id=WHUZ35F63HPQLMT4OHITVNGVRVE35LXZZYA4LICROWXWRRFBL7QQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.528382183 (502h38m46.399465218s) [seg_pos={0 0}, upload=441108]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:07:46.528584508 (502h38m46.399667543s) [noise=true , address=45.91.116.9:4263 , piece_id=6HPT7BHKUWTJV637IC2WWLO5P2QAJUY55MJHIIF2GFPROBBB7JBA, seg_pos={0 0}, upload=441108]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:40618->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:40618->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40618->45.91.116.9:4263: use of closed network connection
Dec 26 14:07:46.528593595 (502h38m46.39967664s) [noise=true , address=45.91.116.9:4263 , piece_id=6HPT7BHKUWTJV637IC2WWLO5P2QAJUY55MJHIIF2GFPROBBB7JBA, seg_pos={0 0}, upload=441108]: Long tail cancellation due to optimal threshold
Dec 26 14:07:46.528937500 (502h38m46.400020535s) [noise=true , address=77.90.31.11:4555 , piece_id=OKWIBDSCGJ57NSWTOAJ3SRZ7MIJQTSXPRGGQNWAIMPKRNN2UEINQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:07:46.528946747 (502h38m46.400029793s) [noise=true , address=77.90.31.11:4555 , piece_id=OKWIBDSCGJ57NSWTOAJ3SRZ7MIJQTSXPRGGQNWAIMPKRNN2UEINQ, seg_pos={0 0}, upload=441108]: Long tail cancellation due to optimal threshold
Dec 26 14:07:46.529092335 (502h38m46.40017537s) [noise=true , address=45.91.116.13:4156 , piece_id=6WSAT4CBID2ECUFAQBNS3H3WCFGFXIAPDOSCVPW5NSNVHOAZEM6Q, seg_pos={0 0}, upload=441108]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 26 14:07:46.529101162 (502h38m46.400184197s) [noise=true , address=45.91.116.13:4156 , piece_id=6WSAT4CBID2ECUFAQBNS3H3WCFGFXIAPDOSCVPW5NSNVHOAZEM6Q, seg_pos={0 0}, upload=441108]: Long tail cancellation due to optimal threshold
Dec 26 14:07:46.529120248 (502h38m46.400203293s) [seg_pos={0 0}, upload=441108]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:07:46.529197134 (502h38m46.400280169s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:07:46.529203747 (502h38m46.400286783s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:07:46.836565970 (502h38m46.707649015s) [upload=441108]: Done uploading segments...
Dec 26 14:07:46.836588884 (502h38m46.707671929s) [upload=441108]: Upload finished. err: <nil>
Duration: 6.043782184s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
17.821102868s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:08:20 UTC)
orbiter warning: context deadline exceeded
Duration: 17.82112969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X......X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..X] duration:
30.002422056s trace:
https://jaeger.dev.storj.io/trace/789322c1f1ca0733
FAIL (2025-12-26 14:08:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: B663A7HFTICDQSVAY6VUOEPOMIPX6Z4RTGODIULBHH7OGLOUHETQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WHUZ35F63HPQLMT4OHITVNGVRVE35LXZZYA4LICROWXWRRFBL7QQ) context deadline exceeded
======LOGS======
Dec 26 14:07:43.883075431 (502h38m43.754158476s) [upload=441108]: Starting upload
Dec 26 14:07:43.883145184 (502h38m43.754228229s) [upload=441108]: Done starting upload
Dec 26 14:07:43.883364942 (502h38m43.754447987s) [upload=441108]: Uploading segments...
Dec 26 14:07:43.883408225 (502h38m43.75449127s) [upload=441108]: Got next segment. Inline: false
Dec 26 14:07:43.883437450 (502h38m43.754520495s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:07:43.883442350 (502h38m43.754525385s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:07:44.135341702 (502h38m44.006424737s) [seg_pos={0 0}, upload=441108]: Begin upload segment...
Dec 26 14:07:44.135396447 (502h38m44.006479492s) [seg_pos={0 0}, upload=441108]: Done begin upload segment.
Dec 26 14:07:44.135421695 (502h38m44.00650473s) [noise=true , address=45.91.116.13:4156 , piece_id=6WSAT4CBID2ECUFAQBNS3H3WCFGFXIAPDOSCVPW5NSNVHOAZEM6Q, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.135437244 (502h38m44.006520289s) [noise=true , address=145.79.227.48:4303 , piece_id=CXXG5UV74SQN4XOP3AURZSEHRSQIZB3YUXPCVHW2MIEX73IHZH3Q, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.135555980 (502h38m44.006639015s) [noise=true , address=45.91.116.5:4297 , piece_id=LYVNAHVXCRNY55EFCPTX6FX2LZV4FTU67COGDVNJH5Q5MJXYD6ZQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.135947646 (502h38m44.007030691s) [noise=true , address=145.79.227.12:4767 , piece_id=EQMJCI5TJDN3TTNW2CYB44SP23AYCWHXRFEN4RBSLGHB725OXB7A, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.136332138 (502h38m44.007415173s) [noise=true , address=145.79.227.19:4077 , piece_id=HIQMV3FDMIHD64YG46FYIDVQ5LE55KD3JLZ6NY2CBL5AP4DXZLVA, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.136741207 (502h38m44.007824252s) [noise=true , address=145.79.227.39:4015 , piece_id=HPHLYNRDJG5ICQP7KYZ7HXBCRUICSTWXMMXMLPCJZ34EHD3VJDFQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.137144695 (502h38m44.00822773s) [noise=true , address=145.79.227.16:4910 , piece_id=NTVBZNP6HRJUQ5IT3IYSQM6J53OLRUVNZE2YF3XLJV2DYQV7WOMA, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.137598339 (502h38m44.008681374s) [noise=true , address=45.91.116.9:4263 , piece_id=6HPT7BHKUWTJV637IC2WWLO5P2QAJUY55MJHIIF2GFPROBBB7JBA, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.138062132 (502h38m44.009145167s) [noise=true , address=77.90.31.10:4519 , piece_id=B663A7HFTICDQSVAY6VUOEPOMIPX6Z4RTGODIULBHH7OGLOUHETQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.138467284 (502h38m44.009550319s) [noise=true , address=145.79.227.14:4840 , piece_id=7D756DMUJM7JE7VVCPMQOU4FY2XXUHNJ7DU5Y6BI5YCXNHGZJFLQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.138882655 (502h38m44.00996569s) [noise=true , address=145.79.227.21:4473 , piece_id=MBRB7BG7EOPY6EJKZZCGQMOJHM72KFRUZD6IQRTN2SP2DKVQV6KQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.139241017 (502h38m44.010324052s) [noise=true , address=145.79.227.15:4875 , piece_id=UHCEXIJU3SQBEJIX24FBZKFVBAIJ62EUPHCF3YFZTAAXZMJ5RDGQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.139306121 (502h38m44.010389156s) [noise=true , address=77.90.31.19:4844 , piece_id=WHUZ35F63HPQLMT4OHITVNGVRVE35LXZZYA4LICROWXWRRFBL7QQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.139718967 (502h38m44.010802002s) [noise=true , address=145.79.227.37:4567 , piece_id=AR4SGG3KYVJ7EONKESDS5HEQ5MT7KPTHWDNZ3MWML5OPJHC6V57Q, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.140132515 (502h38m44.01121556s) [noise=true , address=77.90.31.11:4555 , piece_id=OKWIBDSCGJ57NSWTOAJ3SRZ7MIJQTSXPRGGQNWAIMPKRNN2UEINQ, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:44.140579877 (502h38m44.011662912s) [noise=true , address=145.79.227.50:4374 , piece_id=VQQ2VH6X46ZCXHWGHFS2XIQBSPF6OWS74PEL7FZBFMTEYS52UL5A, seg_pos={0 0}, upload=441108]: Uploading piece...
Dec 26 14:07:45.843911697 (502h38m45.714994732s) [upload=441108]: Next returned nil segment
Dec 26 14:07:45.843932196 (502h38m45.715015241s) [upload=441108]: Waiting for error group managing segments...
Dec 26 14:07:46.127244323 (502h38m45.998327368s) [noise=true , address=145.79.227.15:4875 , piece_id=UHCEXIJU3SQBEJIX24FBZKFVBAIJ62EUPHCF3YFZTAAXZMJ5RDGQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.132567029 (502h38m46.003650084s) [noise=true , address=145.79.227.48:4303 , piece_id=CXXG5UV74SQN4XOP3AURZSEHRSQIZB3YUXPCVHW2MIEX73IHZH3Q, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.138759751 (502h38m46.009842786s) [noise=true , address=145.79.227.37:4567 , piece_id=AR4SGG3KYVJ7EONKESDS5HEQ5MT7KPTHWDNZ3MWML5OPJHC6V57Q, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.141097462 (502h38m46.012180497s) [noise=true , address=145.79.227.14:4840 , piece_id=7D756DMUJM7JE7VVCPMQOU4FY2XXUHNJ7DU5Y6BI5YCXNHGZJFLQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.142037854 (502h38m46.013120889s) [noise=true , address=145.79.227.16:4910 , piece_id=NTVBZNP6HRJUQ5IT3IYSQM6J53OLRUVNZE2YF3XLJV2DYQV7WOMA, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.143284998 (502h38m46.014368033s) [noise=true , address=145.79.227.21:4473 , piece_id=MBRB7BG7EOPY6EJKZZCGQMOJHM72KFRUZD6IQRTN2SP2DKVQV6KQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.147003078 (502h38m46.018086113s) [noise=true , address=145.79.227.12:4767 , piece_id=EQMJCI5TJDN3TTNW2CYB44SP23AYCWHXRFEN4RBSLGHB725OXB7A, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.149266107 (502h38m46.020349152s) [noise=true , address=145.79.227.19:4077 , piece_id=HIQMV3FDMIHD64YG46FYIDVQ5LE55KD3JLZ6NY2CBL5AP4DXZLVA, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.163873552 (502h38m46.034956598s) [noise=true , address=145.79.227.50:4374 , piece_id=VQQ2VH6X46ZCXHWGHFS2XIQBSPF6OWS74PEL7FZBFMTEYS52UL5A, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.175094869 (502h38m46.046177905s) [noise=true , address=145.79.227.39:4015 , piece_id=HPHLYNRDJG5ICQP7KYZ7HXBCRUICSTWXMMXMLPCJZ34EHD3VJDFQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.344919597 (502h38m46.216002642s) [noise=true , address=77.90.31.10:4519 , piece_id=B663A7HFTICDQSVAY6VUOEPOMIPX6Z4RTGODIULBHH7OGLOUHETQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.402876376 (502h38m46.273959411s) [noise=true , address=45.91.116.5:4297 , piece_id=LYVNAHVXCRNY55EFCPTX6FX2LZV4FTU67COGDVNJH5Q5MJXYD6ZQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.528360601 (502h38m46.399443636s) [noise=true , address=77.90.31.19:4844 , piece_id=WHUZ35F63HPQLMT4OHITVNGVRVE35LXZZYA4LICROWXWRRFBL7QQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: <nil>
Dec 26 14:07:46.528382183 (502h38m46.399465218s) [seg_pos={0 0}, upload=441108]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:07:46.528584508 (502h38m46.399667543s) [noise=true , address=45.91.116.9:4263 , piece_id=6HPT7BHKUWTJV637IC2WWLO5P2QAJUY55MJHIIF2GFPROBBB7JBA, seg_pos={0 0}, upload=441108]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:40618->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:40618->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40618->45.91.116.9:4263: use of closed network connection
Dec 26 14:07:46.528593595 (502h38m46.39967664s) [noise=true , address=45.91.116.9:4263 , piece_id=6HPT7BHKUWTJV637IC2WWLO5P2QAJUY55MJHIIF2GFPROBBB7JBA, seg_pos={0 0}, upload=441108]: Long tail cancellation due to optimal threshold
Dec 26 14:07:46.528937500 (502h38m46.400020535s) [noise=true , address=77.90.31.11:4555 , piece_id=OKWIBDSCGJ57NSWTOAJ3SRZ7MIJQTSXPRGGQNWAIMPKRNN2UEINQ, seg_pos={0 0}, upload=441108]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 26 14:07:46.528946747 (502h38m46.400029793s) [noise=true , address=77.90.31.11:4555 , piece_id=OKWIBDSCGJ57NSWTOAJ3SRZ7MIJQTSXPRGGQNWAIMPKRNN2UEINQ, seg_pos={0 0}, upload=441108]: Long tail cancellation due to optimal threshold
Dec 26 14:07:46.529092335 (502h38m46.40017537s) [noise=true , address=45.91.116.13:4156 , piece_id=6WSAT4CBID2ECUFAQBNS3H3WCFGFXIAPDOSCVPW5NSNVHOAZEM6Q, seg_pos={0 0}, upload=441108]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 26 14:07:46.529101162 (502h38m46.400184197s) [noise=true , address=45.91.116.13:4156 , piece_id=6WSAT4CBID2ECUFAQBNS3H3WCFGFXIAPDOSCVPW5NSNVHOAZEM6Q, seg_pos={0 0}, upload=441108]: Long tail cancellation due to optimal threshold
Dec 26 14:07:46.529120248 (502h38m46.400203293s) [seg_pos={0 0}, upload=441108]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:07:46.529197134 (502h38m46.400280169s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:07:46.529203747 (502h38m46.400286783s) [upload=441108]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:07:46.836565970 (502h38m46.707649015s) [upload=441108]: Done uploading segments...
Dec 26 14:07:46.836588884 (502h38m46.707671929s) [upload=441108]: Upload finished. err: <nil>
Duration: 30.002449396s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X........X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X....] duration:
53.459482494s trace:
https://jaeger.dev.storj.io/trace/30de7fc815fa37c4
FAIL (2025-12-26 14:08:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA) context deadline exceeded; piecestore: (Node ID: 1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, Piece ID: W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA) context deadline exceeded; piecestore: (Node ID: 12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, Piece ID: WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ) context deadline exceeded
======LOGS======
[noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130327276 (502h34m30.001410311s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130583704 (502h34m30.001666749s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130953539 (502h34m30.002036574s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131220797 (502h34m30.002303832s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131464200 (502h34m30.002547235s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131703606 (502h34m30.002786641s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132004109 (502h34m30.003087144s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132257551 (502h34m30.003340586s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132559897 (502h34m30.003642932s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132921065 (502h34m30.00400411s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133234351 (502h34m30.004317386s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133531537 (502h34m30.004614583s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133922762 (502h34m30.005005807s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134242982 (502h34m30.005326017s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134646621 (502h34m30.005729656s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:31.511869402 (502h34m31.382952438s) [upload=441053]: Next returned nil segment
Dec 26 14:03:31.511898638 (502h34m31.382981673s) [upload=441053]: Waiting for error group managing segments...
Dec 26 14:03:31.858692107 (502h34m31.729775142s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.033763154 (502h34m31.904846189s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.049074560 (502h34m31.920157595s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.196860500 (502h34m32.067943545s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245048429 (502h34m32.116131474s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245834415 (502h34m32.11691746s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.260889284 (502h34m32.131972339s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.319421619 (502h34m32.190504654s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.371711729 (502h34m32.242794774s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.391762580 (502h34m32.262845625s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.774271347 (502h34m32.645354392s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.938459792 (502h34m32.809542837s) [noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.008572379 (502h34m32.879655424s) [noise=true , address=145.79.227.30:4785 , piece_id=PWKY6OAGQ25CGM4A5KQVHFMUNH2F23GIOERKTJTW45HFAIAEQMYQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014328018 (502h34m32.885411053s) [noise=true , address=145.79.227.44:4215 , piece_id=ECVNPWQAEWX2IU4QHEHUW4F2TLZYGE6NPZ7EBAGASYL5SFNVGBAQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014355661 (502h34m32.885438706s) [seg_pos={0 0}, upload=441053]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:03:33.014598734 (502h34m32.885681779s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection
Dec 26 14:03:33.014607961 (502h34m32.885691026s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014677514 (502h34m32.885760559s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014693144 (502h34m32.885776179s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014851225 (502h34m32.88593426s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014859290 (502h34m32.885942335s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014949702 (502h34m32.886032747s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014959551 (502h34m32.886042596s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014987504 (502h34m32.886070539s) [seg_pos={0 0}, upload=441053]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:03:33.015077776 (502h34m32.886160821s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:03:33.015085050 (502h34m32.886168095s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:03:33.314140235 (502h34m33.18522328s) [upload=441053]: Done uploading segments...
Dec 26 14:03:33.314162848 (502h34m33.185245893s) [upload=441053]: Upload finished. err: <nil>
Duration: 53.459499307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
7.529697224s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:07:57 UTC)
orbiter warning: context deadline exceeded
Duration: 7.529723683s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [X..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXXX] duration:
5.001648372s trace:
https://jaeger.dev.storj.io/trace/377769c1f46d64e9
FAIL (2025-12-26 14:07:54 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001672889s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XX..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXXX] duration:
5.00044939s trace:
https://jaeger.dev.storj.io/trace/73e266c9be6e4954
FAIL (2025-12-26 14:07:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000469498s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
20.634391237s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:07:23 UTC)
orbiter warning: context deadline exceeded
Duration: 20.634417397s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X........X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X...] duration:
47.743587949s trace:
https://jaeger.dev.storj.io/trace/30de7fc815fa37c4
FAIL (2025-12-26 14:07:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA) context deadline exceeded; piecestore: (Node ID: 1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, Piece ID: W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA) context deadline exceeded; piecestore: (Node ID: 12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, Piece ID: WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ) context deadline exceeded
======LOGS======
[noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130327276 (502h34m30.001410311s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130583704 (502h34m30.001666749s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130953539 (502h34m30.002036574s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131220797 (502h34m30.002303832s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131464200 (502h34m30.002547235s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131703606 (502h34m30.002786641s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132004109 (502h34m30.003087144s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132257551 (502h34m30.003340586s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132559897 (502h34m30.003642932s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132921065 (502h34m30.00400411s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133234351 (502h34m30.004317386s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133531537 (502h34m30.004614583s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133922762 (502h34m30.005005807s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134242982 (502h34m30.005326017s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134646621 (502h34m30.005729656s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:31.511869402 (502h34m31.382952438s) [upload=441053]: Next returned nil segment
Dec 26 14:03:31.511898638 (502h34m31.382981673s) [upload=441053]: Waiting for error group managing segments...
Dec 26 14:03:31.858692107 (502h34m31.729775142s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.033763154 (502h34m31.904846189s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.049074560 (502h34m31.920157595s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.196860500 (502h34m32.067943545s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245048429 (502h34m32.116131474s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245834415 (502h34m32.11691746s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.260889284 (502h34m32.131972339s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.319421619 (502h34m32.190504654s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.371711729 (502h34m32.242794774s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.391762580 (502h34m32.262845625s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.774271347 (502h34m32.645354392s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.938459792 (502h34m32.809542837s) [noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.008572379 (502h34m32.879655424s) [noise=true , address=145.79.227.30:4785 , piece_id=PWKY6OAGQ25CGM4A5KQVHFMUNH2F23GIOERKTJTW45HFAIAEQMYQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014328018 (502h34m32.885411053s) [noise=true , address=145.79.227.44:4215 , piece_id=ECVNPWQAEWX2IU4QHEHUW4F2TLZYGE6NPZ7EBAGASYL5SFNVGBAQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014355661 (502h34m32.885438706s) [seg_pos={0 0}, upload=441053]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:03:33.014598734 (502h34m32.885681779s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection
Dec 26 14:03:33.014607961 (502h34m32.885691026s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014677514 (502h34m32.885760559s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014693144 (502h34m32.885776179s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014851225 (502h34m32.88593426s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014859290 (502h34m32.885942335s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014949702 (502h34m32.886032747s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014959551 (502h34m32.886042596s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014987504 (502h34m32.886070539s) [seg_pos={0 0}, upload=441053]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:03:33.015077776 (502h34m32.886160821s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:03:33.015085050 (502h34m32.886168095s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:03:33.314140235 (502h34m33.18522328s) [upload=441053]: Done uploading segments...
Dec 26 14:03:33.314162848 (502h34m33.185245893s) [upload=441053]: Upload finished. err: <nil>
Duration: 47.743606223s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
13.837146424s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:07:07 UTC)
orbiter warning: context deadline exceeded
Duration: 13.837174157s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXX..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXXX] duration:
5.00187869s trace:
https://jaeger.dev.storj.io/trace/4f043a0f4611d4ad
FAIL (2025-12-26 14:06:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00189886s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
13.648555647s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:06:41 UTC)
orbiter warning: context deadline exceeded
Duration: 13.648580935s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X.X......X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X..] duration:
6.110437749s trace:
https://jaeger.dev.storj.io/trace/d664ee695f74c95
FAIL (2025-12-26 14:06:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:04:53.209892877 (502h35m53.080975922s) [upload=441070]: Starting upload
Dec 26 14:04:53.209987277 (502h35m53.081070322s) [upload=441070]: Done starting upload
Dec 26 14:04:53.210051770 (502h35m53.081134805s) [upload=441070]: Uploading segments...
Dec 26 14:04:53.210216412 (502h35m53.081299458s) [upload=441070]: Got next segment. Inline: false
Dec 26 14:04:53.210255828 (502h35m53.081338873s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:04:53.210261990 (502h35m53.081345025s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:04:53.463301653 (502h35m53.334384688s) [seg_pos={0 0}, upload=441070]: Begin upload segment...
Dec 26 14:04:53.463346368 (502h35m53.334429403s) [seg_pos={0 0}, upload=441070]: Done begin upload segment.
Dec 26 14:04:53.463377537 (502h35m53.334460603s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.463394661 (502h35m53.334477696s) [noise=true , address=145.79.227.45:4230 , piece_id=I6NI7PJHTXVEXD4KZT7OMSIC65UEXKW4KDHKK55JEPZGMJT56I4A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.463893139 (502h35m53.334976174s) [noise=true , address=145.79.227.51:4411 , piece_id=OV4D4XWIKXS33XL42XACRG5T5OUHT5YZUW34B2WCI6NT7PJDPHDQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464222707 (502h35m53.335305742s) [noise=true , address=45.91.116.7:4010 , piece_id=MAJQLL36XJ2EML5KVVFDTEZ2CP7IGYGVGI5Y6CSBIHZOPS3M4PUA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464544290 (502h35m53.335627325s) [noise=true , address=145.79.227.23:4546 , piece_id=M5232OELODUZFKPFPSNO2S72LMJMCUEBUNSDH35KQI2MYODTTULQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464860361 (502h35m53.335943406s) [noise=true , address=145.79.227.36:4518 , piece_id=B3B46ZMK7RY5X4NNG4BBK62ATKI5OMVKO3LXFRM2TPRTPP7M2TQA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464943790 (502h35m53.336026835s) [noise=true , address=145.79.227.11:4731 , piece_id=DIAXDQ24Z7LHSAP74N444EIHWYGACWRLZEQ3P2A6ZOL5TUE4LPGA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465175912 (502h35m53.336258947s) [noise=true , address=145.79.227.48:4301 , piece_id=36N43RVWGVMYGZQ4HEYWORLQJTV7BJT2UFULS37NZ3OMD2G4B4QQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465286743 (502h35m53.336369778s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465525147 (502h35m53.336608182s) [noise=true , address=45.91.116.5:4299 , piece_id=PGZJGWAXN5C6Z34JTNITSRNUTP5EK5DUMIU7LIEVU2XRVO2WRLAQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465731691 (502h35m53.336814726s) [noise=true , address=45.91.116.13:4154 , piece_id=63OWTSPBQBEFCYMUYTACG4SNYLBCGP4HFPGASIO2IUMWI5MQNOUQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465881596 (502h35m53.336964631s) [noise=true , address=145.79.227.38:4977 , piece_id=S5L5GMEEFLPNBQZ4I62V47OKVU26CZT7LNGLCHPNE3ZGA5AH7Y3A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466277600 (502h35m53.337360635s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466288952 (502h35m53.337371987s) [noise=true , address=145.79.227.49:4337 , piece_id=IEOZVV7VSG4BC36VTLHF4W5LGEIAW4SAU7GQOUH2TH3ALEDFLK7Q, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466631664 (502h35m53.337714699s) [noise=true , address=145.79.227.35:4482 , piece_id=BA3POICRZY66M6Q344EXAB3PB2SOG2ZVIJQT7BQ2TRW6Z2YRAX4A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466971852 (502h35m53.338054887s) [noise=true , address=145.79.227.37:4567 , piece_id=6MML3OABE6HYX35UDRMDFLSIWD3E7RS5WM7V5FOVYC625US44SHQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:55.183452903 (502h35m55.054535948s) [upload=441070]: Next returned nil segment
Dec 26 14:04:55.183467179 (502h35m55.054550214s) [upload=441070]: Waiting for error group managing segments...
Dec 26 14:04:55.469935727 (502h35m55.341018772s) [noise=true , address=145.79.227.11:4731 , piece_id=DIAXDQ24Z7LHSAP74N444EIHWYGACWRLZEQ3P2A6ZOL5TUE4LPGA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.470920242 (502h35m55.342003287s) [noise=true , address=145.79.227.51:4411 , piece_id=OV4D4XWIKXS33XL42XACRG5T5OUHT5YZUW34B2WCI6NT7PJDPHDQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.471810196 (502h35m55.342893231s) [noise=true , address=145.79.227.38:4977 , piece_id=S5L5GMEEFLPNBQZ4I62V47OKVU26CZT7LNGLCHPNE3ZGA5AH7Y3A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.473043886 (502h35m55.344126921s) [noise=true , address=145.79.227.45:4230 , piece_id=I6NI7PJHTXVEXD4KZT7OMSIC65UEXKW4KDHKK55JEPZGMJT56I4A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.473074042 (502h35m55.344157077s) [noise=true , address=145.79.227.49:4337 , piece_id=IEOZVV7VSG4BC36VTLHF4W5LGEIAW4SAU7GQOUH2TH3ALEDFLK7Q, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.475572550 (502h35m55.346655585s) [noise=true , address=145.79.227.37:4567 , piece_id=6MML3OABE6HYX35UDRMDFLSIWD3E7RS5WM7V5FOVYC625US44SHQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.484435958 (502h35m55.355518993s) [noise=true , address=145.79.227.36:4518 , piece_id=B3B46ZMK7RY5X4NNG4BBK62ATKI5OMVKO3LXFRM2TPRTPP7M2TQA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.488304484 (502h35m55.359387519s) [noise=true , address=145.79.227.35:4482 , piece_id=BA3POICRZY66M6Q344EXAB3PB2SOG2ZVIJQT7BQ2TRW6Z2YRAX4A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.495328750 (502h35m55.366411785s) [noise=true , address=145.79.227.48:4301 , piece_id=36N43RVWGVMYGZQ4HEYWORLQJTV7BJT2UFULS37NZ3OMD2G4B4QQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.497028777 (502h35m55.368111822s) [noise=true , address=145.79.227.23:4546 , piece_id=M5232OELODUZFKPFPSNO2S72LMJMCUEBUNSDH35KQI2MYODTTULQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.714683912 (502h35m55.585766957s) [noise=true , address=45.91.116.7:4010 , piece_id=MAJQLL36XJ2EML5KVVFDTEZ2CP7IGYGVGI5Y6CSBIHZOPS3M4PUA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.733536130 (502h35m55.604619175s) [noise=true , address=45.91.116.5:4299 , piece_id=PGZJGWAXN5C6Z34JTNITSRNUTP5EK5DUMIU7LIEVU2XRVO2WRLAQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.760145318 (502h35m55.631228353s) [noise=true , address=45.91.116.13:4154 , piece_id=63OWTSPBQBEFCYMUYTACG4SNYLBCGP4HFPGASIO2IUMWI5MQNOUQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.760165006 (502h35m55.631248041s) [seg_pos={0 0}, upload=441070]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:04:55.760448095 (502h35m55.63153113s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 26 14:04:55.760459476 (502h35m55.631542511s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760479986 (502h35m55.631563021s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:04:55.760489173 (502h35m55.631572208s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760743919 (502h35m55.631826964s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:46160->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:04:55.760753577 (502h35m55.631836622s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760780178 (502h35m55.631863223s) [seg_pos={0 0}, upload=441070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:04:55.760856262 (502h35m55.631939297s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:04:55.760861212 (502h35m55.631944257s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:04:56.049042020 (502h35m55.920125065s) [upload=441070]: Done uploading segments...
Dec 26 14:04:56.049063411 (502h35m55.920146446s) [upload=441070]: Upload finished. err: <nil>
Duration: 6.110465111s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
14.69919316s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:06:20 UTC)
orbiter warning: context deadline exceeded
Duration: 14.699224399s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
20 out of 60 is failed (threshold=3) [XX.X........X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X..] duration:
48.625921987s trace:
https://jaeger.dev.storj.io/trace/30de7fc815fa37c4
FAIL (2025-12-26 14:06:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA) context deadline exceeded; piecestore: (Node ID: 1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, Piece ID: W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA) context deadline exceeded; piecestore: (Node ID: 12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, Piece ID: WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ) context deadline exceeded
======LOGS======
[noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130327276 (502h34m30.001410311s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130583704 (502h34m30.001666749s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130953539 (502h34m30.002036574s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131220797 (502h34m30.002303832s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131464200 (502h34m30.002547235s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131703606 (502h34m30.002786641s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132004109 (502h34m30.003087144s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132257551 (502h34m30.003340586s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132559897 (502h34m30.003642932s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132921065 (502h34m30.00400411s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133234351 (502h34m30.004317386s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133531537 (502h34m30.004614583s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133922762 (502h34m30.005005807s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134242982 (502h34m30.005326017s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134646621 (502h34m30.005729656s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:31.511869402 (502h34m31.382952438s) [upload=441053]: Next returned nil segment
Dec 26 14:03:31.511898638 (502h34m31.382981673s) [upload=441053]: Waiting for error group managing segments...
Dec 26 14:03:31.858692107 (502h34m31.729775142s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.033763154 (502h34m31.904846189s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.049074560 (502h34m31.920157595s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.196860500 (502h34m32.067943545s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245048429 (502h34m32.116131474s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245834415 (502h34m32.11691746s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.260889284 (502h34m32.131972339s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.319421619 (502h34m32.190504654s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.371711729 (502h34m32.242794774s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.391762580 (502h34m32.262845625s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.774271347 (502h34m32.645354392s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.938459792 (502h34m32.809542837s) [noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.008572379 (502h34m32.879655424s) [noise=true , address=145.79.227.30:4785 , piece_id=PWKY6OAGQ25CGM4A5KQVHFMUNH2F23GIOERKTJTW45HFAIAEQMYQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014328018 (502h34m32.885411053s) [noise=true , address=145.79.227.44:4215 , piece_id=ECVNPWQAEWX2IU4QHEHUW4F2TLZYGE6NPZ7EBAGASYL5SFNVGBAQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014355661 (502h34m32.885438706s) [seg_pos={0 0}, upload=441053]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:03:33.014598734 (502h34m32.885681779s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection
Dec 26 14:03:33.014607961 (502h34m32.885691026s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014677514 (502h34m32.885760559s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014693144 (502h34m32.885776179s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014851225 (502h34m32.88593426s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014859290 (502h34m32.885942335s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014949702 (502h34m32.886032747s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014959551 (502h34m32.886042596s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014987504 (502h34m32.886070539s) [seg_pos={0 0}, upload=441053]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:03:33.015077776 (502h34m32.886160821s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:03:33.015085050 (502h34m32.886168095s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:03:33.314140235 (502h34m33.18522328s) [upload=441053]: Done uploading segments...
Dec 26 14:03:33.314162848 (502h34m33.185245893s) [upload=441053]: Upload finished. err: <nil>
Duration: 48.625940383s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X.X......X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X.] duration:
6.177164039s trace:
https://jaeger.dev.storj.io/trace/d664ee695f74c95
FAIL (2025-12-26 14:06:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:04:53.209892877 (502h35m53.080975922s) [upload=441070]: Starting upload
Dec 26 14:04:53.209987277 (502h35m53.081070322s) [upload=441070]: Done starting upload
Dec 26 14:04:53.210051770 (502h35m53.081134805s) [upload=441070]: Uploading segments...
Dec 26 14:04:53.210216412 (502h35m53.081299458s) [upload=441070]: Got next segment. Inline: false
Dec 26 14:04:53.210255828 (502h35m53.081338873s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:04:53.210261990 (502h35m53.081345025s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:04:53.463301653 (502h35m53.334384688s) [seg_pos={0 0}, upload=441070]: Begin upload segment...
Dec 26 14:04:53.463346368 (502h35m53.334429403s) [seg_pos={0 0}, upload=441070]: Done begin upload segment.
Dec 26 14:04:53.463377537 (502h35m53.334460603s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.463394661 (502h35m53.334477696s) [noise=true , address=145.79.227.45:4230 , piece_id=I6NI7PJHTXVEXD4KZT7OMSIC65UEXKW4KDHKK55JEPZGMJT56I4A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.463893139 (502h35m53.334976174s) [noise=true , address=145.79.227.51:4411 , piece_id=OV4D4XWIKXS33XL42XACRG5T5OUHT5YZUW34B2WCI6NT7PJDPHDQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464222707 (502h35m53.335305742s) [noise=true , address=45.91.116.7:4010 , piece_id=MAJQLL36XJ2EML5KVVFDTEZ2CP7IGYGVGI5Y6CSBIHZOPS3M4PUA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464544290 (502h35m53.335627325s) [noise=true , address=145.79.227.23:4546 , piece_id=M5232OELODUZFKPFPSNO2S72LMJMCUEBUNSDH35KQI2MYODTTULQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464860361 (502h35m53.335943406s) [noise=true , address=145.79.227.36:4518 , piece_id=B3B46ZMK7RY5X4NNG4BBK62ATKI5OMVKO3LXFRM2TPRTPP7M2TQA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464943790 (502h35m53.336026835s) [noise=true , address=145.79.227.11:4731 , piece_id=DIAXDQ24Z7LHSAP74N444EIHWYGACWRLZEQ3P2A6ZOL5TUE4LPGA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465175912 (502h35m53.336258947s) [noise=true , address=145.79.227.48:4301 , piece_id=36N43RVWGVMYGZQ4HEYWORLQJTV7BJT2UFULS37NZ3OMD2G4B4QQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465286743 (502h35m53.336369778s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465525147 (502h35m53.336608182s) [noise=true , address=45.91.116.5:4299 , piece_id=PGZJGWAXN5C6Z34JTNITSRNUTP5EK5DUMIU7LIEVU2XRVO2WRLAQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465731691 (502h35m53.336814726s) [noise=true , address=45.91.116.13:4154 , piece_id=63OWTSPBQBEFCYMUYTACG4SNYLBCGP4HFPGASIO2IUMWI5MQNOUQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465881596 (502h35m53.336964631s) [noise=true , address=145.79.227.38:4977 , piece_id=S5L5GMEEFLPNBQZ4I62V47OKVU26CZT7LNGLCHPNE3ZGA5AH7Y3A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466277600 (502h35m53.337360635s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466288952 (502h35m53.337371987s) [noise=true , address=145.79.227.49:4337 , piece_id=IEOZVV7VSG4BC36VTLHF4W5LGEIAW4SAU7GQOUH2TH3ALEDFLK7Q, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466631664 (502h35m53.337714699s) [noise=true , address=145.79.227.35:4482 , piece_id=BA3POICRZY66M6Q344EXAB3PB2SOG2ZVIJQT7BQ2TRW6Z2YRAX4A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466971852 (502h35m53.338054887s) [noise=true , address=145.79.227.37:4567 , piece_id=6MML3OABE6HYX35UDRMDFLSIWD3E7RS5WM7V5FOVYC625US44SHQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:55.183452903 (502h35m55.054535948s) [upload=441070]: Next returned nil segment
Dec 26 14:04:55.183467179 (502h35m55.054550214s) [upload=441070]: Waiting for error group managing segments...
Dec 26 14:04:55.469935727 (502h35m55.341018772s) [noise=true , address=145.79.227.11:4731 , piece_id=DIAXDQ24Z7LHSAP74N444EIHWYGACWRLZEQ3P2A6ZOL5TUE4LPGA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.470920242 (502h35m55.342003287s) [noise=true , address=145.79.227.51:4411 , piece_id=OV4D4XWIKXS33XL42XACRG5T5OUHT5YZUW34B2WCI6NT7PJDPHDQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.471810196 (502h35m55.342893231s) [noise=true , address=145.79.227.38:4977 , piece_id=S5L5GMEEFLPNBQZ4I62V47OKVU26CZT7LNGLCHPNE3ZGA5AH7Y3A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.473043886 (502h35m55.344126921s) [noise=true , address=145.79.227.45:4230 , piece_id=I6NI7PJHTXVEXD4KZT7OMSIC65UEXKW4KDHKK55JEPZGMJT56I4A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.473074042 (502h35m55.344157077s) [noise=true , address=145.79.227.49:4337 , piece_id=IEOZVV7VSG4BC36VTLHF4W5LGEIAW4SAU7GQOUH2TH3ALEDFLK7Q, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.475572550 (502h35m55.346655585s) [noise=true , address=145.79.227.37:4567 , piece_id=6MML3OABE6HYX35UDRMDFLSIWD3E7RS5WM7V5FOVYC625US44SHQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.484435958 (502h35m55.355518993s) [noise=true , address=145.79.227.36:4518 , piece_id=B3B46ZMK7RY5X4NNG4BBK62ATKI5OMVKO3LXFRM2TPRTPP7M2TQA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.488304484 (502h35m55.359387519s) [noise=true , address=145.79.227.35:4482 , piece_id=BA3POICRZY66M6Q344EXAB3PB2SOG2ZVIJQT7BQ2TRW6Z2YRAX4A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.495328750 (502h35m55.366411785s) [noise=true , address=145.79.227.48:4301 , piece_id=36N43RVWGVMYGZQ4HEYWORLQJTV7BJT2UFULS37NZ3OMD2G4B4QQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.497028777 (502h35m55.368111822s) [noise=true , address=145.79.227.23:4546 , piece_id=M5232OELODUZFKPFPSNO2S72LMJMCUEBUNSDH35KQI2MYODTTULQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.714683912 (502h35m55.585766957s) [noise=true , address=45.91.116.7:4010 , piece_id=MAJQLL36XJ2EML5KVVFDTEZ2CP7IGYGVGI5Y6CSBIHZOPS3M4PUA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.733536130 (502h35m55.604619175s) [noise=true , address=45.91.116.5:4299 , piece_id=PGZJGWAXN5C6Z34JTNITSRNUTP5EK5DUMIU7LIEVU2XRVO2WRLAQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.760145318 (502h35m55.631228353s) [noise=true , address=45.91.116.13:4154 , piece_id=63OWTSPBQBEFCYMUYTACG4SNYLBCGP4HFPGASIO2IUMWI5MQNOUQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.760165006 (502h35m55.631248041s) [seg_pos={0 0}, upload=441070]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:04:55.760448095 (502h35m55.63153113s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 26 14:04:55.760459476 (502h35m55.631542511s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760479986 (502h35m55.631563021s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:04:55.760489173 (502h35m55.631572208s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760743919 (502h35m55.631826964s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:46160->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:04:55.760753577 (502h35m55.631836622s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760780178 (502h35m55.631863223s) [seg_pos={0 0}, upload=441070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:04:55.760856262 (502h35m55.631939297s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:04:55.760861212 (502h35m55.631944257s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:04:56.049042020 (502h35m55.920125065s) [upload=441070]: Done uploading segments...
Dec 26 14:04:56.049063411 (502h35m55.920146446s) [upload=441070]: Upload finished. err: <nil>
Duration: 6.177195507s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
17.775225729s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:05:55 UTC)
orbiter warning: context deadline exceeded
Duration: 17.775249104s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXXX..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXXX] duration:
5.001159127s trace:
https://jaeger.dev.storj.io/trace/760a13ea136ecae7
FAIL (2025-12-26 14:05:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00118124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
14.731120083s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:05:39 UTC)
orbiter warning: context deadline exceeded
Duration: 14.731144239s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
20 out of 60 is failed (threshold=3) [.XX.X........X.XXX.X.X..X.X.X.XX.....X..........X..X..X.X.X.] duration:
53.158599877s trace:
https://jaeger.dev.storj.io/trace/30de7fc815fa37c4
FAIL (2025-12-26 14:05:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA) context deadline exceeded; piecestore: (Node ID: 1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, Piece ID: W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA) context deadline exceeded; piecestore: (Node ID: 12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, Piece ID: WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ) context deadline exceeded
======LOGS======
[noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130327276 (502h34m30.001410311s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130583704 (502h34m30.001666749s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.130953539 (502h34m30.002036574s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131220797 (502h34m30.002303832s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131464200 (502h34m30.002547235s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.131703606 (502h34m30.002786641s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132004109 (502h34m30.003087144s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132257551 (502h34m30.003340586s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132559897 (502h34m30.003642932s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.132921065 (502h34m30.00400411s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133234351 (502h34m30.004317386s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133531537 (502h34m30.004614583s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.133922762 (502h34m30.005005807s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134242982 (502h34m30.005326017s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:30.134646621 (502h34m30.005729656s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Uploading piece...
Dec 26 14:03:31.511869402 (502h34m31.382952438s) [upload=441053]: Next returned nil segment
Dec 26 14:03:31.511898638 (502h34m31.382981673s) [upload=441053]: Waiting for error group managing segments...
Dec 26 14:03:31.858692107 (502h34m31.729775142s) [noise=true , address=64.49.13.137:4495 , piece_id=HW2FFKAVX7HN6VA5QO2MNBZT6TGHID5BI4XXB33XRERVWVTOTQ5Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.033763154 (502h34m31.904846189s) [noise=true , address=145.79.227.28:4738 , piece_id=VV42PGSQO2VK2BMTSNGOBEC5EPORTG2MDGWBZ2FZPYBHRZNJJ7WQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.049074560 (502h34m31.920157595s) [noise=true , address=145.79.227.52:4458 , piece_id=SDEFEQMZNZPRFRMDDD2LKRPP5HZRXCYQHJF56LTKLPLCHUZX367Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.196860500 (502h34m32.067943545s) [noise=true , address=77.90.31.14:4674 , piece_id=WFDDLK2OA6AJUDZDT7HJEWVMQTD3H4YFDPBZVCTKZMGZDUMUDAXA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245048429 (502h34m32.116131474s) [noise=true , address=77.90.31.12:4623 , piece_id=W7W4B56OCFJ4CRMRRWN3XAKK4JHN4DQBKADYPSBDC4OYTSNVRIDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.245834415 (502h34m32.11691746s) [noise=true , address=77.90.31.13:4630 , piece_id=4UTJLU7NJMMFIWJSYPRL4M5VOIJZ5DTERUCL3UELV6V7Z76O5OHQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.260889284 (502h34m32.131972339s) [noise=true , address=77.90.31.11:4583 , piece_id=GMOPKOF2UBSP4P44KHKYYIBOD2ZREJ6VPKWS6UEBSFSKW3T3O4QQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.319421619 (502h34m32.190504654s) [noise=true , address=77.90.31.15:4723 , piece_id=EDE6OZQCE4BBKNA4TWMBJ7XTI22KBROK4BQAM7YGCWBTBCSV5APA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.371711729 (502h34m32.242794774s) [noise=true , address=45.91.116.14:4204 , piece_id=ODEGOQ74C2DVZ5MKX3GEJA62N3NT2ZVVZACKWFZEP6USWEE3BSUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.391762580 (502h34m32.262845625s) [noise=true , address=45.91.116.10:4100 , piece_id=WYFKQSAC5SPUWCB3UOBL37LBAQQCR5O3AHC2ZFEU5XXGQMDFINRQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.774271347 (502h34m32.645354392s) [noise=true , address=193.149.164.143:4695 , piece_id=WWTFFCXEFXGFV3A6GMVKWPBSY4G7NFRSZFIA3MEB2UAOGXO63MDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:32.938459792 (502h34m32.809542837s) [noise=true , address=145.79.227.43:4161 , piece_id=RDPQH63MNOXNDNGEFIZLOSQEV4S5YQZGNNFXY73KIBAYXOKQJ23Q, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.008572379 (502h34m32.879655424s) [noise=true , address=145.79.227.30:4785 , piece_id=PWKY6OAGQ25CGM4A5KQVHFMUNH2F23GIOERKTJTW45HFAIAEQMYQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014328018 (502h34m32.885411053s) [noise=true , address=145.79.227.44:4215 , piece_id=ECVNPWQAEWX2IU4QHEHUW4F2TLZYGE6NPZ7EBAGASYL5SFNVGBAQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: <nil>
Dec 26 14:03:33.014355661 (502h34m32.885438706s) [seg_pos={0 0}, upload=441053]: Segment reached optimal threshold of 14 pieces.
Dec 26 14:03:33.014598734 (502h34m32.885681779s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43906->77.90.31.16:4768: use of closed network connection
Dec 26 14:03:33.014607961 (502h34m32.885691026s) [noise=true , address=77.90.31.16:4768 , piece_id=BOHMZESICGTJQA7QZ7N7GY76KXD3VOHRLJMTX2JQWDZXFSUVUOCA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014677514 (502h34m32.885760559s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014693144 (502h34m32.885776179s) [noise=true , address=77.90.31.19:4871 , piece_id=HZPJ254DCCNMQEXNI7UFYEAKKHTQRAAPMX3I7CZORZG3NNSXRZUA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014851225 (502h34m32.88593426s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014859290 (502h34m32.885942335s) [noise=true , address=145.79.227.40:4063 , piece_id=CJYFECIVQMXHQCJIJ4M45DXC4XZF2CKSRAPHTDCENK6FA7ZC6SDA, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014949702 (502h34m32.886032747s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP): protocol: expected piece hash; context canceled; EOF
Dec 26 14:03:33.014959551 (502h34m32.886042596s) [noise=true , address=45.91.116.9:4265 , piece_id=KZOJGHZ4QBAQC3LNHB7T5XYDUUIXJKN4PGEIIGW73GODMTUCOCXQ, seg_pos={0 0}, upload=441053]: Long tail cancellation due to optimal threshold
Dec 26 14:03:33.014987504 (502h34m32.886070539s) [seg_pos={0 0}, upload=441053]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 26 14:03:33.015077776 (502h34m32.886160821s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:03:33.015085050 (502h34m32.886168095s) [upload=441053]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:03:33.314140235 (502h34m33.18522328s) [upload=441053]: Done uploading segments...
Dec 26 14:03:33.314162848 (502h34m33.185245893s) [upload=441053]: Upload finished. err: <nil>
Duration: 53.158629604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXXXX..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXXX] duration:
5.001068928s trace:
https://jaeger.dev.storj.io/trace/77b6cdf89994f5a3
FAIL (2025-12-26 14:05:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001092944s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X.X.X......X.X.XX.X.X..X.XXXXX.X..XX..XX...X...X...........X] duration:
30.00510488s trace:
https://jaeger.dev.storj.io/trace/d664ee695f74c95
FAIL (2025-12-26 14:05:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 14:04:53.209892877 (502h35m53.080975922s) [upload=441070]: Starting upload
Dec 26 14:04:53.209987277 (502h35m53.081070322s) [upload=441070]: Done starting upload
Dec 26 14:04:53.210051770 (502h35m53.081134805s) [upload=441070]: Uploading segments...
Dec 26 14:04:53.210216412 (502h35m53.081299458s) [upload=441070]: Got next segment. Inline: false
Dec 26 14:04:53.210255828 (502h35m53.081338873s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 14:04:53.210261990 (502h35m53.081345025s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 14:04:53.463301653 (502h35m53.334384688s) [seg_pos={0 0}, upload=441070]: Begin upload segment...
Dec 26 14:04:53.463346368 (502h35m53.334429403s) [seg_pos={0 0}, upload=441070]: Done begin upload segment.
Dec 26 14:04:53.463377537 (502h35m53.334460603s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.463394661 (502h35m53.334477696s) [noise=true , address=145.79.227.45:4230 , piece_id=I6NI7PJHTXVEXD4KZT7OMSIC65UEXKW4KDHKK55JEPZGMJT56I4A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.463893139 (502h35m53.334976174s) [noise=true , address=145.79.227.51:4411 , piece_id=OV4D4XWIKXS33XL42XACRG5T5OUHT5YZUW34B2WCI6NT7PJDPHDQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464222707 (502h35m53.335305742s) [noise=true , address=45.91.116.7:4010 , piece_id=MAJQLL36XJ2EML5KVVFDTEZ2CP7IGYGVGI5Y6CSBIHZOPS3M4PUA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464544290 (502h35m53.335627325s) [noise=true , address=145.79.227.23:4546 , piece_id=M5232OELODUZFKPFPSNO2S72LMJMCUEBUNSDH35KQI2MYODTTULQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464860361 (502h35m53.335943406s) [noise=true , address=145.79.227.36:4518 , piece_id=B3B46ZMK7RY5X4NNG4BBK62ATKI5OMVKO3LXFRM2TPRTPP7M2TQA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.464943790 (502h35m53.336026835s) [noise=true , address=145.79.227.11:4731 , piece_id=DIAXDQ24Z7LHSAP74N444EIHWYGACWRLZEQ3P2A6ZOL5TUE4LPGA, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465175912 (502h35m53.336258947s) [noise=true , address=145.79.227.48:4301 , piece_id=36N43RVWGVMYGZQ4HEYWORLQJTV7BJT2UFULS37NZ3OMD2G4B4QQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465286743 (502h35m53.336369778s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465525147 (502h35m53.336608182s) [noise=true , address=45.91.116.5:4299 , piece_id=PGZJGWAXN5C6Z34JTNITSRNUTP5EK5DUMIU7LIEVU2XRVO2WRLAQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465731691 (502h35m53.336814726s) [noise=true , address=45.91.116.13:4154 , piece_id=63OWTSPBQBEFCYMUYTACG4SNYLBCGP4HFPGASIO2IUMWI5MQNOUQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.465881596 (502h35m53.336964631s) [noise=true , address=145.79.227.38:4977 , piece_id=S5L5GMEEFLPNBQZ4I62V47OKVU26CZT7LNGLCHPNE3ZGA5AH7Y3A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466277600 (502h35m53.337360635s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466288952 (502h35m53.337371987s) [noise=true , address=145.79.227.49:4337 , piece_id=IEOZVV7VSG4BC36VTLHF4W5LGEIAW4SAU7GQOUH2TH3ALEDFLK7Q, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466631664 (502h35m53.337714699s) [noise=true , address=145.79.227.35:4482 , piece_id=BA3POICRZY66M6Q344EXAB3PB2SOG2ZVIJQT7BQ2TRW6Z2YRAX4A, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:53.466971852 (502h35m53.338054887s) [noise=true , address=145.79.227.37:4567 , piece_id=6MML3OABE6HYX35UDRMDFLSIWD3E7RS5WM7V5FOVYC625US44SHQ, seg_pos={0 0}, upload=441070]: Uploading piece...
Dec 26 14:04:55.183452903 (502h35m55.054535948s) [upload=441070]: Next returned nil segment
Dec 26 14:04:55.183467179 (502h35m55.054550214s) [upload=441070]: Waiting for error group managing segments...
Dec 26 14:04:55.469935727 (502h35m55.341018772s) [noise=true , address=145.79.227.11:4731 , piece_id=DIAXDQ24Z7LHSAP74N444EIHWYGACWRLZEQ3P2A6ZOL5TUE4LPGA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.470920242 (502h35m55.342003287s) [noise=true , address=145.79.227.51:4411 , piece_id=OV4D4XWIKXS33XL42XACRG5T5OUHT5YZUW34B2WCI6NT7PJDPHDQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.471810196 (502h35m55.342893231s) [noise=true , address=145.79.227.38:4977 , piece_id=S5L5GMEEFLPNBQZ4I62V47OKVU26CZT7LNGLCHPNE3ZGA5AH7Y3A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.473043886 (502h35m55.344126921s) [noise=true , address=145.79.227.45:4230 , piece_id=I6NI7PJHTXVEXD4KZT7OMSIC65UEXKW4KDHKK55JEPZGMJT56I4A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.473074042 (502h35m55.344157077s) [noise=true , address=145.79.227.49:4337 , piece_id=IEOZVV7VSG4BC36VTLHF4W5LGEIAW4SAU7GQOUH2TH3ALEDFLK7Q, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.475572550 (502h35m55.346655585s) [noise=true , address=145.79.227.37:4567 , piece_id=6MML3OABE6HYX35UDRMDFLSIWD3E7RS5WM7V5FOVYC625US44SHQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.484435958 (502h35m55.355518993s) [noise=true , address=145.79.227.36:4518 , piece_id=B3B46ZMK7RY5X4NNG4BBK62ATKI5OMVKO3LXFRM2TPRTPP7M2TQA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.488304484 (502h35m55.359387519s) [noise=true , address=145.79.227.35:4482 , piece_id=BA3POICRZY66M6Q344EXAB3PB2SOG2ZVIJQT7BQ2TRW6Z2YRAX4A, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.495328750 (502h35m55.366411785s) [noise=true , address=145.79.227.48:4301 , piece_id=36N43RVWGVMYGZQ4HEYWORLQJTV7BJT2UFULS37NZ3OMD2G4B4QQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.497028777 (502h35m55.368111822s) [noise=true , address=145.79.227.23:4546 , piece_id=M5232OELODUZFKPFPSNO2S72LMJMCUEBUNSDH35KQI2MYODTTULQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.714683912 (502h35m55.585766957s) [noise=true , address=45.91.116.7:4010 , piece_id=MAJQLL36XJ2EML5KVVFDTEZ2CP7IGYGVGI5Y6CSBIHZOPS3M4PUA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.733536130 (502h35m55.604619175s) [noise=true , address=45.91.116.5:4299 , piece_id=PGZJGWAXN5C6Z34JTNITSRNUTP5EK5DUMIU7LIEVU2XRVO2WRLAQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.760145318 (502h35m55.631228353s) [noise=true , address=45.91.116.13:4154 , piece_id=63OWTSPBQBEFCYMUYTACG4SNYLBCGP4HFPGASIO2IUMWI5MQNOUQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: <nil>
Dec 26 14:04:55.760165006 (502h35m55.631248041s) [seg_pos={0 0}, upload=441070]: Segment reached optimal threshold of 13 pieces.
Dec 26 14:04:55.760448095 (502h35m55.63153113s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 26 14:04:55.760459476 (502h35m55.631542511s) [noise=true , address=77.90.31.15:4700 , piece_id=7ZRXCYBFJHFFHJTJJEFEEDDKZJX7UWM52LVGTZZK3BLQ4KT34Q7Q, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760479986 (502h35m55.631563021s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 26 14:04:55.760489173 (502h35m55.631572208s) [noise=true , address=45.91.116.11:4119 , piece_id=3W5ZW6SE5ZGPJHFGABZ7RLYTLXTLGG3KBH4E6X3HVIJFVABJOFNA, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760743919 (502h35m55.631826964s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: read tcp 188.166.225.170:46160->45.91.116.9:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 26 14:04:55.760753577 (502h35m55.631836622s) [noise=true , address=45.91.116.9:4264 , piece_id=PFFWJEBTSAJ2HULC2NQX2JDK33EL6HQ3DLWQOX7J4KHQNLNUIJLQ, seg_pos={0 0}, upload=441070]: Long tail cancellation due to optimal threshold
Dec 26 14:04:55.760780178 (502h35m55.631863223s) [seg_pos={0 0}, upload=441070]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 14:04:55.760856262 (502h35m55.631939297s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 14:04:55.760861212 (502h35m55.631944257s) [upload=441070]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 14:04:56.049042020 (502h35m55.920125065s) [upload=441070]: Done uploading segments...
Dec 26 14:04:56.049063411 (502h35m55.920146446s) [upload=441070]: Upload finished. err: <nil>
Duration: 30.005128997s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
13.135463965s trace:
https://jaeger.dev.storj.io/trace/5b63846e68ea1b18
FAIL (2025-12-26 14:05:08 UTC)
orbiter warning: context deadline exceeded
Duration: 13.135517547s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
8.661204077s trace:
https://jaeger.dev.storj.io/trace/5500efce92c20bf8
FAIL (2025-12-26 14:05:07 UTC)
orbiter warning: context deadline exceeded
Duration: 8.661238833s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [.XXXXX..X.X.X.X...XXXX.XX.X.XXX..XX.X.XX.XX.XXXXXXXXXXXXXXXX] duration:
5.001247548s trace:
https://jaeger.dev.storj.io/trace/17328823312a7a76
FAIL (2025-12-26 14:04:48 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001267987s
core
uplink
us-select-1
us1-placement-20
|