Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..XX..XX....X.....XXX....X.....X..XX....XX.X...X.X..........] duration:
6.085045703s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 07:00:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 6.085081261s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [..X.X..X.X.X............X..X......X.XX..X.X..X.XX.......X.X.] duration:
3.427335316s trace:
https://jaeger.dev.storj.io/trace/74ae39b5feeabe1d
FAIL (2025-12-31 06:59:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.427362217s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.....X.X..........................] duration:
8.840333636s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:59:29 UTC)
orbiter warning: context deadline exceeded
Duration: 8.840391565s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X...............X..............X.............] duration:
42.311227166s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:59:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 42.311245632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...XX..XX....X.....XXX....X.....X..XX....XX.X...X.X.........] duration:
5.868727732s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:59:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.868756356s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [...X.X..X.X.X............X..X......X.XX..X.X..X.XX.......X.X] duration:
5.000875064s trace:
https://jaeger.dev.storj.io/trace/74ae39b5feeabe1d
FAIL (2025-12-31 06:59:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000898107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................X.....X.X.........................] duration:
6.13251062s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:58:55 UTC)
orbiter warning: context deadline exceeded
Duration: 6.132539144s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X...............X..............X............] duration:
25.19800614s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:58:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 25.198035707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X........] duration:
6.007350434s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:58:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 6.007370873s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [....X.X..X.X.X............X..X......X.XX..X.X..X.XX.......X.] duration:
3.535743543s trace:
https://jaeger.dev.storj.io/trace/555f7550c47f5fe4
FAIL (2025-12-31 06:58:12 UTC)
orbiter warning: context deadline exceeded
Duration: 3.535787437s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.....X.X........................] duration:
14.338239724s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:58:07 UTC)
orbiter warning: context deadline exceeded
Duration: 14.338266626s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.....X.X..X.X.X............X..X......X.XX..X.X..X.XX.......X] duration:
5.001069713s trace:
https://jaeger.dev.storj.io/trace/555f7550c47f5fe4
FAIL (2025-12-31 06:57:41 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001097404s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X.......] duration:
5.983833018s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:57:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.983861793s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X...............X..............X...........] duration:
16.350130897s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:57:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 16.350155293s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X......] duration:
6.218722449s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:57:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 6.218751744s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X...............X..............X..........] duration:
39.370871538s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:57:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 39.370898317s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................X.....X.X.......................] duration:
11.276500233s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:57:07 UTC)
orbiter warning: context deadline exceeded
Duration: 11.276547623s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X.....] duration:
5.806203651s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:56:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.806236023s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [......X.X..X.X.X............X..X......X.XX..X.X..X.XX.......] duration:
3.905139988s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:56:44 UTC)
orbiter warning: context deadline exceeded
Duration: 3.905164908s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X....] duration:
5.739043384s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:56:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.739075076s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................X.....X.X......................] duration:
8.986696095s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:56:14 UTC)
orbiter warning: context deadline exceeded
Duration: 8.986722324s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................X...............X..............X.........] duration:
19.300133982s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:55:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 19.300150793s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X...............X..............X........] duration:
37.360916211s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:55:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 37.361084391s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.......X.X..X.X.X............X..X......X.XX..X.X..X.XX......] duration:
4.497218211s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:55:30 UTC)
orbiter warning: context deadline exceeded
Duration: 4.497275181s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X...] duration:
5.847954889s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:55:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.84798163s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................X.....X.X.....................] duration:
13.089236368s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:55:17 UTC)
orbiter warning: context deadline exceeded
Duration: 13.089267156s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X...............X..............X.......] duration:
25.084786274s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:54:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 25.084802816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [........X.X..X.X.X............X..X......X.XX..X.X..X.XX.....] duration:
3.811777906s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:54:33 UTC)
orbiter warning: context deadline exceeded
Duration: 3.811804485s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X..] duration:
5.75630678s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:54:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.756336987s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X...............X..............X......] duration:
30.771965239s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:54:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 30.771995866s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................X.....X.X....................] duration:
5.805596562s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:54:14 UTC)
orbiter warning: context deadline exceeded
Duration: 5.805624726s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X.] duration:
5.987143251s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:53:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 5.98716977s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.........X.X..X.X.X............X..X......X.XX..X.X..X.XX....] duration:
3.511494969s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:53:57 UTC)
orbiter warning: context deadline exceeded
Duration: 3.511525759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................X.....X.X...................] duration:
5.897285587s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:53:37 UTC)
orbiter warning: context deadline exceeded
Duration: 5.897307109s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...X.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.X] duration:
30.003225094s trace:
https://jaeger.dev.storj.io/trace/1d5e07dfc9c7daf8
FAIL (2025-12-31 06:53:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: 6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ) context deadline exceeded
======LOGS======
ad=539288]: Uploading segments...
Dec 31 06:52:58.073262824 (615h23m57.944345889s) [upload=539288]: Got next segment. Inline: false
Dec 31 06:52:58.073336333 (615h23m57.944419368s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:52:58.073343908 (615h23m57.944426943s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:52:58.319135661 (615h23m58.190218696s) [seg_pos={0 0}, upload=539288]: Begin upload segment...
Dec 31 06:52:58.319186347 (615h23m58.190269382s) [seg_pos={0 0}, upload=539288]: Done begin upload segment.
Dec 31 06:52:58.319211695 (615h23m58.19029473s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.319356812 (615h23m58.190439857s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320179038 (615h23m58.191262073s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.320498997 (615h23m58.191582032s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321059895 (615h23m58.19214294s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321600454 (615h23m58.192683489s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.321819712 (615h23m58.192902747s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322045281 (615h23m58.193128316s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322267224 (615h23m58.193350259s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322560241 (615h23m58.193643286s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.322650444 (615h23m58.193733479s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323251969 (615h23m58.194335004s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323521723 (615h23m58.194604758s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323801565 (615h23m58.1948846s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.323979845 (615h23m58.19506289s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:52:58.324367613 (615h23m58.195450658s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Uploading piece...
Dec 31 06:53:00.028408864 (615h23m59.899491899s) [upload=539288]: Next returned nil segment
Dec 31 06:53:00.028427239 (615h23m59.899510284s) [upload=539288]: Waiting for error group managing segments...
Dec 31 06:53:00.309331772 (615h24m0.180414827s) [noise=true , address=145.79.227.39:4013 , piece_id=QMS5K2JT2QMKMHBQDXYYUNNL33PRECV2JVNWMEOHE5EH73DMKA6Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.309833818 (615h24m0.180916863s) [noise=true , address=145.79.227.46:4602 , piece_id=CTJ5JZUX2MENECYILGSNWWYGHDKT7QIQ6YNAT2X3GF7NRU73CO3Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.320947380 (615h24m0.192030415s) [noise=true , address=145.79.227.7:4294 , piece_id=62CKBGTNP6XS2KGAFQ7KHLN6OPUT7JIXTKIM5JLU6EKNVKD6RHYQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.324266350 (615h24m0.195349385s) [noise=true , address=145.79.227.4:4186 , piece_id=M2IRSNVKTA3RBF2A7RNULCEWGP3YENXUZ36EIWK4XGSQTIP4JHFQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.326705414 (615h24m0.197788459s) [noise=true , address=145.79.227.52:4447 , piece_id=63JXORDAGN77CDGYFNPQK5JTLZNVTCZ2W4XCTOUA3HTUN6AMR5AA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.327940446 (615h24m0.199023481s) [noise=true , address=145.79.227.26:4653 , piece_id=RWFQDYS62DBUDZWEVNOJMRKSEDUZGDEVDG5XIK7YH56VEJXGYX4Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.330259582 (615h24m0.201342627s) [noise=true , address=145.79.227.37:4567 , piece_id=KA4XLKZREQWG5573CXHIOHHAHZYIKNAK2AJOIBDKT6JRDLZRL53Q, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.341350029 (615h24m0.212433085s) [noise=true , address=145.79.227.47:4266 , piece_id=KKWH7HEIS7EM2GWAT6BSH4BG2F6K6H2V7KD7ROTAXUGXYCRFYEEA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.498126635 (615h24m0.36920968s) [noise=true , address=77.90.31.21:4880 , piece_id=4DH5CE6FAMB3E5FMD47K2HKUI2IPXCNTFJRGE7LTULA4WJC7GMFA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.505262653 (615h24m0.376345688s) [noise=true , address=77.90.31.19:4842 , piece_id=EEA2FMZ4Z3FTE7Q6YXWV6FX4WUFUMVHCQLLNC3WPH6RPZCOVKLVA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.587184143 (615h24m0.458267178s) [noise=true , address=77.90.31.15:4697 , piece_id=6W2CHKWOMPWHDMPMHR3IBM5YYJ22SYWN622BFXKWHG6UMA43MYNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.636776475 (615h24m0.50785952s) [noise=true , address=77.90.31.10:4519 , piece_id=LX47QODK7CPFAMXT7QQVXRNOZ3QMDAWTQOSQXVJEIKK4OUSBQIPQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821687211 (615h24m0.692770246s) [noise=true , address=77.90.31.12:4589 , piece_id=6CMJFYWNWCNFDBIEGJ435G3ZXMU3XA2CX5JNPSCSGKOAMM6OYLRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: <nil>
Dec 31 06:53:00.821712169 (615h24m0.692795204s) [seg_pos={0 0}, upload=539288]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:53:00.822063618 (615h24m0.693146653s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39920->45.91.116.10:4081: use of closed network connection
Dec 31 06:53:00.822074148 (615h24m0.693157203s) [noise=true , address=45.91.116.10:4081 , piece_id=KDHMCWW4NYMR7PD2LBXJEEADMT5NVSBXRGESXPQGWEGDGPUSNHNA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822282424 (615h24m0.693365459s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46554->45.91.116.14:4191: use of closed network connection
Dec 31 06:53:00.822304466 (615h24m0.693387512s) [noise=true , address=45.91.116.14:4191 , piece_id=YPGIEATUT5I4KP24ASSGFMT4BMXLBYXQPTTPXWERXVWKPPBJ3NMA, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822323162 (615h24m0.693406207s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF
Dec 31 06:53:00.822365283 (615h24m0.693448328s) [noise=true , address=45.91.116.7:4011 , piece_id=XIKU73VBQFUL2HJFZTJGB2DVHIJZLM2UPADLNHZGBW5OF3Q2NHRQ, seg_pos={0 0}, upload=539288]: Long tail cancellation due to optimal threshold
Dec 31 06:53:00.822405128 (615h24m0.693488173s) [seg_pos={0 0}, upload=539288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:53:00.822542099 (615h24m0.693625144s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:53:00.822549393 (615h24m0.693632438s) [upload=539288]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:53:01.114211430 (615h24m0.985294465s) [upload=539288]: Done uploading segments...
Dec 31 06:53:01.114230987 (615h24m0.985314032s) [upload=539288]: Upload finished. err: <nil>
Duration: 30.003278516s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X......................X...............X..............X.....] duration:
25.223567602s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:53:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 25.223592979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [..........X.X..X.X.X............X..X......X.XX..X.X..X.XX...] duration:
3.582944062s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:52:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.583010439s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X......................X...............X..............X....] duration:
15.320430801s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:52:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 15.320465699s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................X.....X.X..................] duration:
6.014562727s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:52:34 UTC)
orbiter warning: context deadline exceeded
Duration: 6.01459596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X.] duration:
5.719963235s trace:
https://jaeger.dev.storj.io/trace/6f8bb32abcb2c245
FAIL (2025-12-31 06:52:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 06:51:10.871000264 (615h22m10.742083309s) [upload=539262]: Starting upload
Dec 31 06:51:10.871090516 (615h22m10.742173561s) [upload=539262]: Done starting upload
Dec 31 06:51:10.871150160 (615h22m10.742233195s) [upload=539262]: Uploading segments...
Dec 31 06:51:10.871346453 (615h22m10.742429498s) [upload=539262]: Got next segment. Inline: false
Dec 31 06:51:10.871418119 (615h22m10.742501164s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:51:10.871424773 (615h22m10.742507808s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:51:11.119131762 (615h22m10.990214807s) [seg_pos={0 0}, upload=539262]: Begin upload segment...
Dec 31 06:51:11.119172821 (615h22m10.990255856s) [seg_pos={0 0}, upload=539262]: Done begin upload segment.
Dec 31 06:51:11.119196645 (615h22m10.99027968s) [noise=true , address=145.79.227.8:4329 , piece_id=VAZNVBKV3OUCXWC5IBNEXM2BA6IEFNYURX52YCNLXAF573AXSMIQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.119207046 (615h22m10.990290081s) [noise=true , address=45.91.116.5:4299 , piece_id=SWIMUSWA64KHOBJIIJWJBZL6SWHZPP4K2VLSC7KEFE66ROYQKYGA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.119275315 (615h22m10.99035835s) [noise=true , address=77.90.31.11:4556 , piece_id=QVQ67RXJ47UOLHOJAEP4CYXIPXU2BPDGEPCO63FRCBN5ZHYZKFZA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.119811446 (615h22m10.990894481s) [noise=true , address=45.91.116.13:4156 , piece_id=4BEQ2665AFMAFCALG33ZU2YSSKIVRSZLD4FGOM2KEKEPXYGZ67CQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.120134882 (615h22m10.991217917s) [noise=true , address=77.90.31.14:4663 , piece_id=NFLE6QKTFMRY7XVMIPAJOVEI6GPATEW4QVD3ISPDCYDBCWLRVE3A, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.120502452 (615h22m10.991585497s) [noise=true , address=45.91.116.6:4975 , piece_id=ISC66HXLCLYHIRZZ3ITVGFBSZGHTKICQYVNBIOS4EZWJA5KF72QQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.120987004 (615h22m10.992070039s) [noise=true , address=145.79.227.28:4727 , piece_id=DGOZREX2B6EMP4RRSPCUQYNAEXLJLKHFYSWD7JIXL6AGD362ZJVA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.121509118 (615h22m10.992592163s) [noise=true , address=45.91.116.10:4084 , piece_id=FLRT2GJPUWWO4YWFSRXDJ644F4ZTIMPE7EGICTFKV42AYDPG644Q, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.122850884 (615h22m10.993933919s) [noise=true , address=77.90.31.18:4805 , piece_id=TG3Z77RHUE7EV3NFVZQ27CWAMLVCWN6IF2JXQK6EOGKGRQBDVBIA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.123452469 (615h22m10.994535504s) [noise=true , address=77.90.31.21:4880 , piece_id=2JF4WDGI3VZZIAHWSH76X65ZEOGVGR52RGPJI5TOOXHBDPILGCXA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.124043815 (615h22m10.99512686s) [noise=true , address=145.79.227.33:4870 , piece_id=PR5MR6TK2H5M3MWU3LSQQKEUCLP5PM6URCHJTKHWLLN54KRJXGCA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.125326518 (615h22m10.996409553s) [noise=true , address=45.91.116.14:4191 , piece_id=ITPLLXK7ZMKUR7PGVXXNQTP5XEH4KZTVMLPYS4CUKQVFRWEPIJKQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.125428592 (615h22m10.996511627s) [noise=true , address=145.79.227.39:4014 , piece_id=XWDFPCMNX7QMXDFEUEPYUQJEZPRMF7KSM4V2XD4T3Y3FJE3TPGGA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.127129120 (615h22m10.998212155s) [noise=true , address=145.79.227.2:4113 , piece_id=6KKQE6NUEQXZXCFQJB2UJCQUHVOMKKPTH4I7MY7LKH3E2SUJZ5QA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.128769293 (615h22m10.999852338s) [noise=true , address=145.79.227.11:4729 , piece_id=I2VOAY7SHXYSAC2C5YELKQUZZISABN446B3EKRZ5RCRL56ZAL6RQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.130686234 (615h22m11.001769269s) [noise=true , address=145.79.227.50:4374 , piece_id=3BIKUS3FEZ4O3ZYNWM27JGPGW3BTOV2JTO2LJVNA37QWLU66Z4MQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:12.801772904 (615h22m12.672855949s) [upload=539262]: Next returned nil segment
Dec 31 06:51:12.801838578 (615h22m12.672921623s) [upload=539262]: Waiting for error group managing segments...
Dec 31 06:51:13.118963220 (615h22m12.990046266s) [noise=true , address=145.79.227.2:4113 , piece_id=6KKQE6NUEQXZXCFQJB2UJCQUHVOMKKPTH4I7MY7LKH3E2SUJZ5QA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.124001143 (615h22m12.995084188s) [noise=true , address=145.79.227.50:4374 , piece_id=3BIKUS3FEZ4O3ZYNWM27JGPGW3BTOV2JTO2LJVNA37QWLU66Z4MQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.124161418 (615h22m12.995244463s) [noise=true , address=145.79.227.28:4727 , piece_id=DGOZREX2B6EMP4RRSPCUQYNAEXLJLKHFYSWD7JIXL6AGD362ZJVA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.130394537 (615h22m13.001477572s) [noise=true , address=145.79.227.11:4729 , piece_id=I2VOAY7SHXYSAC2C5YELKQUZZISABN446B3EKRZ5RCRL56ZAL6RQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.131522885 (615h22m13.002605921s) [noise=true , address=145.79.227.39:4014 , piece_id=XWDFPCMNX7QMXDFEUEPYUQJEZPRMF7KSM4V2XD4T3Y3FJE3TPGGA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.158072570 (615h22m13.029155605s) [noise=true , address=145.79.227.33:4870 , piece_id=PR5MR6TK2H5M3MWU3LSQQKEUCLP5PM6URCHJTKHWLLN54KRJXGCA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.161520936 (615h22m13.032603981s) [noise=true , address=145.79.227.8:4329 , piece_id=VAZNVBKV3OUCXWC5IBNEXM2BA6IEFNYURX52YCNLXAF573AXSMIQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.295943146 (615h22m13.167026181s) [noise=true , address=77.90.31.11:4556 , piece_id=QVQ67RXJ47UOLHOJAEP4CYXIPXU2BPDGEPCO63FRCBN5ZHYZKFZA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.306886864 (615h22m13.177969909s) [noise=true , address=77.90.31.18:4805 , piece_id=TG3Z77RHUE7EV3NFVZQ27CWAMLVCWN6IF2JXQK6EOGKGRQBDVBIA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.484831767 (615h22m13.355914802s) [noise=true , address=45.91.116.6:4975 , piece_id=ISC66HXLCLYHIRZZ3ITVGFBSZGHTKICQYVNBIOS4EZWJA5KF72QQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.919026346 (615h22m13.790109391s) [noise=true , address=45.91.116.10:4084 , piece_id=FLRT2GJPUWWO4YWFSRXDJ644F4ZTIMPE7EGICTFKV42AYDPG644Q, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.987819789 (615h22m13.858902834s) [noise=true , address=77.90.31.21:4880 , piece_id=2JF4WDGI3VZZIAHWSH76X65ZEOGVGR52RGPJI5TOOXHBDPILGCXA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:14.096704782 (615h22m13.967787827s) [noise=true , address=77.90.31.14:4663 , piece_id=NFLE6QKTFMRY7XVMIPAJOVEI6GPATEW4QVD3ISPDCYDBCWLRVE3A, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:14.096729258 (615h22m13.967812293s) [seg_pos={0 0}, upload=539262]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:51:14.096948716 (615h22m13.968031761s) [noise=true , address=45.91.116.13:4156 , piece_id=4BEQ2665AFMAFCALG33ZU2YSSKIVRSZLD4FGOM2KEKEPXYGZ67CQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 31 06:51:14.096957373 (615h22m13.968040418s) [noise=true , address=45.91.116.13:4156 , piece_id=4BEQ2665AFMAFCALG33ZU2YSSKIVRSZLD4FGOM2KEKEPXYGZ67CQ, seg_pos={0 0}, upload=539262]: Long tail cancellation due to optimal threshold
Dec 31 06:51:14.097325713 (615h22m13.968408758s) [noise=true , address=45.91.116.5:4299 , piece_id=SWIMUSWA64KHOBJIIJWJBZL6SWHZPP4K2VLSC7KEFE66ROYQKYGA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 31 06:51:14.097347335 (615h22m13.96843037s) [noise=true , address=45.91.116.5:4299 , piece_id=SWIMUSWA64KHOBJIIJWJBZL6SWHZPP4K2VLSC7KEFE66ROYQKYGA, seg_pos={0 0}, upload=539262]: Long tail cancellation due to optimal threshold
Dec 31 06:51:14.097372964 (615h22m13.968455999s) [noise=true , address=45.91.116.14:4191 , piece_id=ITPLLXK7ZMKUR7PGVXXNQTP5XEH4KZTVMLPYS4CUKQVFRWEPIJKQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 31 06:51:14.097380528 (615h22m13.968463563s) [noise=true , address=45.91.116.14:4191 , piece_id=ITPLLXK7ZMKUR7PGVXXNQTP5XEH4KZTVMLPYS4CUKQVFRWEPIJKQ, seg_pos={0 0}, upload=539262]: Long tail cancellation due to optimal threshold
Dec 31 06:51:14.097413521 (615h22m13.968496556s) [seg_pos={0 0}, upload=539262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:51:14.097521988 (615h22m13.968605023s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:51:14.097528440 (615h22m13.968611485s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:51:14.383894874 (615h22m14.254977919s) [upload=539262]: Done uploading segments...
Dec 31 06:51:14.383919021 (615h22m14.255002066s) [upload=539262]: Upload finished. err: <nil>
Duration: 5.719979475s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [...........X.X..X.X.X............X..X......X.XX..X.X..X.XX..] duration:
4.068208086s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:52:08 UTC)
orbiter warning: context deadline exceeded
Duration: 4.068250325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X......................X...............X..............X...] duration:
36.992719737s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:52:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 36.992751046s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [............X.X..X.X.X............X..X......X.XX..X.X..X.XX.] duration:
3.734278446s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:51:44 UTC)
orbiter warning: context deadline exceeded
Duration: 3.734310135s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.....X.XX.....XX..XX....X.....XXX....X.....X..XX....XX.X...X] duration:
30.002371621s trace:
https://jaeger.dev.storj.io/trace/6f8bb32abcb2c245
FAIL (2025-12-31 06:51:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 06:51:10.871000264 (615h22m10.742083309s) [upload=539262]: Starting upload
Dec 31 06:51:10.871090516 (615h22m10.742173561s) [upload=539262]: Done starting upload
Dec 31 06:51:10.871150160 (615h22m10.742233195s) [upload=539262]: Uploading segments...
Dec 31 06:51:10.871346453 (615h22m10.742429498s) [upload=539262]: Got next segment. Inline: false
Dec 31 06:51:10.871418119 (615h22m10.742501164s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 06:51:10.871424773 (615h22m10.742507808s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 06:51:11.119131762 (615h22m10.990214807s) [seg_pos={0 0}, upload=539262]: Begin upload segment...
Dec 31 06:51:11.119172821 (615h22m10.990255856s) [seg_pos={0 0}, upload=539262]: Done begin upload segment.
Dec 31 06:51:11.119196645 (615h22m10.99027968s) [noise=true , address=145.79.227.8:4329 , piece_id=VAZNVBKV3OUCXWC5IBNEXM2BA6IEFNYURX52YCNLXAF573AXSMIQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.119207046 (615h22m10.990290081s) [noise=true , address=45.91.116.5:4299 , piece_id=SWIMUSWA64KHOBJIIJWJBZL6SWHZPP4K2VLSC7KEFE66ROYQKYGA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.119275315 (615h22m10.99035835s) [noise=true , address=77.90.31.11:4556 , piece_id=QVQ67RXJ47UOLHOJAEP4CYXIPXU2BPDGEPCO63FRCBN5ZHYZKFZA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.119811446 (615h22m10.990894481s) [noise=true , address=45.91.116.13:4156 , piece_id=4BEQ2665AFMAFCALG33ZU2YSSKIVRSZLD4FGOM2KEKEPXYGZ67CQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.120134882 (615h22m10.991217917s) [noise=true , address=77.90.31.14:4663 , piece_id=NFLE6QKTFMRY7XVMIPAJOVEI6GPATEW4QVD3ISPDCYDBCWLRVE3A, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.120502452 (615h22m10.991585497s) [noise=true , address=45.91.116.6:4975 , piece_id=ISC66HXLCLYHIRZZ3ITVGFBSZGHTKICQYVNBIOS4EZWJA5KF72QQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.120987004 (615h22m10.992070039s) [noise=true , address=145.79.227.28:4727 , piece_id=DGOZREX2B6EMP4RRSPCUQYNAEXLJLKHFYSWD7JIXL6AGD362ZJVA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.121509118 (615h22m10.992592163s) [noise=true , address=45.91.116.10:4084 , piece_id=FLRT2GJPUWWO4YWFSRXDJ644F4ZTIMPE7EGICTFKV42AYDPG644Q, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.122850884 (615h22m10.993933919s) [noise=true , address=77.90.31.18:4805 , piece_id=TG3Z77RHUE7EV3NFVZQ27CWAMLVCWN6IF2JXQK6EOGKGRQBDVBIA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.123452469 (615h22m10.994535504s) [noise=true , address=77.90.31.21:4880 , piece_id=2JF4WDGI3VZZIAHWSH76X65ZEOGVGR52RGPJI5TOOXHBDPILGCXA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.124043815 (615h22m10.99512686s) [noise=true , address=145.79.227.33:4870 , piece_id=PR5MR6TK2H5M3MWU3LSQQKEUCLP5PM6URCHJTKHWLLN54KRJXGCA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.125326518 (615h22m10.996409553s) [noise=true , address=45.91.116.14:4191 , piece_id=ITPLLXK7ZMKUR7PGVXXNQTP5XEH4KZTVMLPYS4CUKQVFRWEPIJKQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.125428592 (615h22m10.996511627s) [noise=true , address=145.79.227.39:4014 , piece_id=XWDFPCMNX7QMXDFEUEPYUQJEZPRMF7KSM4V2XD4T3Y3FJE3TPGGA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.127129120 (615h22m10.998212155s) [noise=true , address=145.79.227.2:4113 , piece_id=6KKQE6NUEQXZXCFQJB2UJCQUHVOMKKPTH4I7MY7LKH3E2SUJZ5QA, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.128769293 (615h22m10.999852338s) [noise=true , address=145.79.227.11:4729 , piece_id=I2VOAY7SHXYSAC2C5YELKQUZZISABN446B3EKRZ5RCRL56ZAL6RQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:11.130686234 (615h22m11.001769269s) [noise=true , address=145.79.227.50:4374 , piece_id=3BIKUS3FEZ4O3ZYNWM27JGPGW3BTOV2JTO2LJVNA37QWLU66Z4MQ, seg_pos={0 0}, upload=539262]: Uploading piece...
Dec 31 06:51:12.801772904 (615h22m12.672855949s) [upload=539262]: Next returned nil segment
Dec 31 06:51:12.801838578 (615h22m12.672921623s) [upload=539262]: Waiting for error group managing segments...
Dec 31 06:51:13.118963220 (615h22m12.990046266s) [noise=true , address=145.79.227.2:4113 , piece_id=6KKQE6NUEQXZXCFQJB2UJCQUHVOMKKPTH4I7MY7LKH3E2SUJZ5QA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.124001143 (615h22m12.995084188s) [noise=true , address=145.79.227.50:4374 , piece_id=3BIKUS3FEZ4O3ZYNWM27JGPGW3BTOV2JTO2LJVNA37QWLU66Z4MQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.124161418 (615h22m12.995244463s) [noise=true , address=145.79.227.28:4727 , piece_id=DGOZREX2B6EMP4RRSPCUQYNAEXLJLKHFYSWD7JIXL6AGD362ZJVA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.130394537 (615h22m13.001477572s) [noise=true , address=145.79.227.11:4729 , piece_id=I2VOAY7SHXYSAC2C5YELKQUZZISABN446B3EKRZ5RCRL56ZAL6RQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.131522885 (615h22m13.002605921s) [noise=true , address=145.79.227.39:4014 , piece_id=XWDFPCMNX7QMXDFEUEPYUQJEZPRMF7KSM4V2XD4T3Y3FJE3TPGGA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.158072570 (615h22m13.029155605s) [noise=true , address=145.79.227.33:4870 , piece_id=PR5MR6TK2H5M3MWU3LSQQKEUCLP5PM6URCHJTKHWLLN54KRJXGCA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.161520936 (615h22m13.032603981s) [noise=true , address=145.79.227.8:4329 , piece_id=VAZNVBKV3OUCXWC5IBNEXM2BA6IEFNYURX52YCNLXAF573AXSMIQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.295943146 (615h22m13.167026181s) [noise=true , address=77.90.31.11:4556 , piece_id=QVQ67RXJ47UOLHOJAEP4CYXIPXU2BPDGEPCO63FRCBN5ZHYZKFZA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.306886864 (615h22m13.177969909s) [noise=true , address=77.90.31.18:4805 , piece_id=TG3Z77RHUE7EV3NFVZQ27CWAMLVCWN6IF2JXQK6EOGKGRQBDVBIA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.484831767 (615h22m13.355914802s) [noise=true , address=45.91.116.6:4975 , piece_id=ISC66HXLCLYHIRZZ3ITVGFBSZGHTKICQYVNBIOS4EZWJA5KF72QQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.919026346 (615h22m13.790109391s) [noise=true , address=45.91.116.10:4084 , piece_id=FLRT2GJPUWWO4YWFSRXDJ644F4ZTIMPE7EGICTFKV42AYDPG644Q, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:13.987819789 (615h22m13.858902834s) [noise=true , address=77.90.31.21:4880 , piece_id=2JF4WDGI3VZZIAHWSH76X65ZEOGVGR52RGPJI5TOOXHBDPILGCXA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:14.096704782 (615h22m13.967787827s) [noise=true , address=77.90.31.14:4663 , piece_id=NFLE6QKTFMRY7XVMIPAJOVEI6GPATEW4QVD3ISPDCYDBCWLRVE3A, seg_pos={0 0}, upload=539262]: Done uploading piece. err: <nil>
Dec 31 06:51:14.096729258 (615h22m13.967812293s) [seg_pos={0 0}, upload=539262]: Segment reached optimal threshold of 13 pieces.
Dec 31 06:51:14.096948716 (615h22m13.968031761s) [noise=true , address=45.91.116.13:4156 , piece_id=4BEQ2665AFMAFCALG33ZU2YSSKIVRSZLD4FGOM2KEKEPXYGZ67CQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 31 06:51:14.096957373 (615h22m13.968040418s) [noise=true , address=45.91.116.13:4156 , piece_id=4BEQ2665AFMAFCALG33ZU2YSSKIVRSZLD4FGOM2KEKEPXYGZ67CQ, seg_pos={0 0}, upload=539262]: Long tail cancellation due to optimal threshold
Dec 31 06:51:14.097325713 (615h22m13.968408758s) [noise=true , address=45.91.116.5:4299 , piece_id=SWIMUSWA64KHOBJIIJWJBZL6SWHZPP4K2VLSC7KEFE66ROYQKYGA, seg_pos={0 0}, upload=539262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 31 06:51:14.097347335 (615h22m13.96843037s) [noise=true , address=45.91.116.5:4299 , piece_id=SWIMUSWA64KHOBJIIJWJBZL6SWHZPP4K2VLSC7KEFE66ROYQKYGA, seg_pos={0 0}, upload=539262]: Long tail cancellation due to optimal threshold
Dec 31 06:51:14.097372964 (615h22m13.968455999s) [noise=true , address=45.91.116.14:4191 , piece_id=ITPLLXK7ZMKUR7PGVXXNQTP5XEH4KZTVMLPYS4CUKQVFRWEPIJKQ, seg_pos={0 0}, upload=539262]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 31 06:51:14.097380528 (615h22m13.968463563s) [noise=true , address=45.91.116.14:4191 , piece_id=ITPLLXK7ZMKUR7PGVXXNQTP5XEH4KZTVMLPYS4CUKQVFRWEPIJKQ, seg_pos={0 0}, upload=539262]: Long tail cancellation due to optimal threshold
Dec 31 06:51:14.097413521 (615h22m13.968496556s) [seg_pos={0 0}, upload=539262]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 06:51:14.097521988 (615h22m13.968605023s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:51:14.097528440 (615h22m13.968611485s) [upload=539262]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:51:14.383894874 (615h22m14.254977919s) [upload=539262]: Done uploading segments...
Dec 31 06:51:14.383919021 (615h22m14.255002066s) [upload=539262]: Upload finished. err: <nil>
Duration: 30.002398983s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................X.....X.X.................] duration:
6.129589236s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:51:38 UTC)
orbiter warning: context deadline exceeded
Duration: 6.12961334s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.............X.X..X.X.X............X..X......X.XX..X.X..X.XX] duration:
5.00131878s trace:
https://jaeger.dev.storj.io/trace/702aa953323c29fb
FAIL (2025-12-31 06:51:27 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001341834s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X......................X...............X..............X..] duration:
18.955158143s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:51:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 18.955188039s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................X.....X.X................] duration:
9.61422008s trace:
https://jaeger.dev.storj.io/trace/10ae64ceddff8b11
FAIL (2025-12-31 06:51:01 UTC)
orbiter warning: context deadline exceeded
Duration: 9.614248875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [X.............X.X..X.X.X............X..X......X.XX..X.X..X.X] duration:
5.001237319s trace:
https://jaeger.dev.storj.io/trace/7b8b51f5ec078b8e
FAIL (2025-12-31 06:50:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001265313s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X......................X...............X..............X.] duration:
41.714874767s trace:
https://jaeger.dev.storj.io/trace/4c7b5edd7a1547c
FAIL (2025-12-31 06:50:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556508594 (615h19m55.427591639s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556650175 (615h19m55.42773321s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556776104 (615h19m55.427859149s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.556989049 (615h19m55.428072084s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557036811 (615h19m55.428119846s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557420571 (615h19m55.428503606s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557481246 (615h19m55.428564291s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557744999 (615h19m55.428828034s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.557754427 (615h19m55.428837462s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558089675 (615h19m55.42917271s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558133378 (615h19m55.429216413s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558413552 (615h19m55.429496597s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.558480349 (615h19m55.429563384s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:55.559316020 (615h19m55.430399065s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Uploading piece...
Dec 31 06:48:56.997849416 (615h19m56.868932451s) [upload=539234]: Next returned nil segment
Dec 31 06:48:56.997869624 (615h19m56.868952669s) [upload=539234]: Waiting for error group managing segments...
Dec 31 06:48:57.408482080 (615h19m57.279565125s) [noise=true , address=145.79.227.2:4141 , piece_id=MJBSKLMTYPRVR6EXVOIT3WALYRTTE7J23IH47LJGWRHYVZUZOWVQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.412096252 (615h19m57.283179287s) [noise=true , address=145.79.227.6:4261 , piece_id=LQYWHJQUWDZQRZBIDZLVLJXV42DKGO2WGXNYUHEMDTO63HVWLIMA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.417132902 (615h19m57.288215947s) [noise=true , address=145.79.227.18:4027 , piece_id=E7PFFQWIFDMB7PAOPYUH7MNPGL2DNVMOUIAOO3LJOW4YEGJD3I3A, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.430260259 (615h19m57.301343294s) [noise=true , address=145.79.227.14:4869 , piece_id=CYJEUX5GLPA7JSG2LXM7F5367ICMSKEDVACCHPMBCLVNNFIIARIQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.510211798 (615h19m57.381294833s) [noise=true , address=145.79.227.34:4931 , piece_id=5RM46WSFGMSNPYSPCNNVIHB5XKJQVABBWQ55BMBD4ME7UENP7QNQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.658031408 (615h19m57.529114453s) [noise=true , address=77.90.31.17:4786 , piece_id=ZCTP32D3OIDV76N7G7CJM53YTHBUL5Z7SQIDMH7J2XVAUKDT3TQQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.699696515 (615h19m57.57077956s) [noise=true , address=77.90.31.10:4529 , piece_id=IZWS5BL72G3GHH5SW6R35ELP4G3WRFZHWI2YZWNM2BRM2J2J6PPA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.857127276 (615h19m57.728210321s) [noise=true , address=145.79.227.35:4508 , piece_id=526QNNPHYBE5BYK4EILMGX4ESPB55ZKXLR4SY3JXPZLW52DJQUEA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:57.867435324 (615h19m57.738518359s) [noise=true , address=45.91.116.12:4240 , piece_id=MCA2S46KK7DJR2UND6HXX6I4XB7FON4BLD7WUV5QFF2F6ZGBPXLA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.265377253 (615h19m58.136460288s) [noise=true , address=77.90.31.18:4814 , piece_id=34Q5HUBMHH4M4WV3US7U7LLJFJEJU524GNSCWB4ZPQK2D4C37PGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.307148061 (615h19m58.178231106s) [noise=true , address=145.79.227.29:4953 , piece_id=WDB2R2ENH3XIQKPWYDFYT3FMBOPBP6BPRYXHW2CKGOPDCIYNKM2Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.310344256 (615h19m58.181427301s) [noise=true , address=145.79.227.30:4782 , piece_id=WDT5BLMZT4WVIUWUUIHBY66BC7AKOQOVSI34UDDWLQWYU76XPW4Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.318250661 (615h19m58.189333696s) [noise=true , address=145.79.227.44:4219 , piece_id=M6J3MT753GLXEK7EXSDCWIUF457J5RPCCN57RF3PTV5J3J3ZRP6Q, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384797335 (615h19m58.25588038s) [noise=true , address=145.79.227.40:4062 , piece_id=I7O7GPP5BY6DVY7MNRXRSZLF5DX277S6HQZKOQDN6L7FNLILGWLQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: <nil>
Dec 31 06:48:58.384826400 (615h19m58.255909445s) [seg_pos={0 0}, upload=539234]: Segment reached optimal threshold of 14 pieces.
Dec 31 06:48:58.385155738 (615h19m58.256238783s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12iQ1hDK2j81J8RpWCFQ8uGk8aPgU5AVJAMdTbstBAA5E6QcqAp): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385166398 (615h19m58.256249433s) [noise=true , address=45.91.116.13:4167 , piece_id=UX2JNJT2Y3Q4JKGYDSBH3DZYFXM3JYXOI6PT2EQO5KDNTFY3NMZQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385247502 (615h19m58.256330547s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai): protocol: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46854->77.90.31.14:4674: use of closed network connection
Dec 31 06:48:58.385272179 (615h19m58.256355214s) [noise=true , address=77.90.31.14:4674 , piece_id=G4IVHA6S2P6P77HJCSWI7RJYRJAP55UKBX53QZ5SWU74O2SLVFSQ, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385343936 (615h19m58.256426971s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF
Dec 31 06:48:58.385352923 (615h19m58.256435968s) [noise=true , address=145.79.227.49:4367 , piece_id=EPV5HYXYSJWPENHYUKNZB6F6Z3ZESM6IWOE5QYJMY5YR26JE3LGA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385517136 (615h19m58.256600181s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45354->64.49.13.169:4710: use of closed network connection
Dec 31 06:48:58.385525962 (615h19m58.256609007s) [noise=true , address=64.49.13.169:4710 , piece_id=TNK6Y5Y2VMFKBXHTTUYVDD37DPTV5BE6DTGWDLLJCRGKR62DNBKA, seg_pos={0 0}, upload=539234]: Long tail cancellation due to optimal threshold
Dec 31 06:48:58.385605985 (615h19m58.25668902s) [seg_pos={0 0}, upload=539234]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 06:48:58.385712628 (615h19m58.256795663s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 06:48:58.385739919 (615h19m58.256822964s) [upload=539234]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 06:48:58.675777653 (615h19m58.546860699s) [upload=539234]: Done uploading segments...
Dec 31 06:48:58.675802391 (615h19m58.546885436s) [upload=539234]: Upload finished. err: <nil>
Duration: 41.714929149s
core
uplink
us-select-1
us1-placement-20
|